20:04:12.044 [INFO - main] (RandomUtils.java:46) Using random seed: 1541707452044 .20:04:12.044 [INFO - main] (KuduTestHarness.java:137) Creating a new MiniKuduCluster... 20:04:12.044 [INFO - main] (MiniKuduCluster.java:146) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskRgtgLL/test-tmp 20:04:12.045 [INFO - main] (KuduBinaryLocator.java:52) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/debug/bin 20:04:12.045 [INFO - main] (MiniKuduCluster.java:197) Starting process: [/tmp/dist-test-taskRgtgLL/java/build/dist-test/../../../build/debug/bin/kudu, test, mini_cluster, --serialization=pb] 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.114217 14752 external_mini_cluster.cc:845] Running /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --block_manager=log 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=localhost 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_ca_key_size=1024 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tsk_num_rsa_bits=512 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.62:45145 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.62 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/logs 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data/info.pb 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_reuseport=true 20:04:12.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --master_addresses=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 with env {} 20:04:12.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) WARNING: Logging before InitGoogleLogging() is written to STDERR 20:04:12.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.159755 14759 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 20:04:12.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.159937 14759 flags.cc:406] Enabled unsafe flag: --never_fsync=true 20:04:12.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.160835 14759 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 20:04:12.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.160866 14759 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 20:04:12.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.160877 14759 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 20:04:12.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.160890 14759 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.162384 14759 master_main.cc:73] Master server non-default flags: 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_ca_key_size=1024 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --master_addresses=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tsk_num_rsa_bits=512 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.62:45145 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_reuseport=true 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports=true 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data/info.pb 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.62 20:04:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync=true 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --heap_profile_path=/tmp/kudu-master.14759 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags=true 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags=true 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/logs 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr=true 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Master server version: 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) kudu 1.9.0-SNAPSHOT 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) revision 94e447869fb2e786919410e6a986beac4e6ef49d 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) build type DEBUG 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) built by ghenke at 05 Nov 2018 12:54:17 PST on ve0518.halxg.cloudera.com 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.162696 14759 mem_tracker.cc:79] Creating tracker root 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.162832 14759 mem_tracker.cc:79] Creating tracker server->root 20:04:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.163331 14759 mem_tracker.cc:79] Creating tracker result-tracker->server->root 20:04:12.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.165338 14759 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.165515 14759 mem_tracker.cc:79] Creating tracker code_cache-sharded_lru_cache->root 20:04:12.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.167181 14759 master_main.cc:80] Initializing master server... 20:04:12.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.167253 14759 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.167287 14759 mem_tracker.cc:79] Creating tracker block_cache-sharded_lru_cache->root 20:04:12.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.167685 14759 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 109960us 20:04:12.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.168501 14759 fs_manager.cc:263] Metadata directory not provided 20:04:12.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.168555 14759 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 20:04:12.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.168567 14759 fs_manager.cc:289] WAL root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal 20:04:12.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.168579 14759 fs_manager.cc:290] Metadata root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal 20:04:12.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.168627 14759 fs_manager.cc:291] Data roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data 20:04:12.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.168639 14759 fs_manager.cc:293] All roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal 20:04:12.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.168853 14759 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file 20:04:12.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.168879 14759 server_base.cc:433] Attempting to create new FS layout instead 20:04:12.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.170492 14759 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data/instance: 20:04:12.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" 20:04:12.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.170795 14759 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal/instance: 20:04:12.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" 20:04:12.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.173426 14759 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.173624 14759 pb_util.cc:291] Reading PB with version 2 starting at offset 597 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.175972 14759 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176107 14759 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176160 14759 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176242 14759 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176282 14759 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176560 14759 mem_tracker.cc:79] Creating tracker log_block_manager->server->root 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176640 14759 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176801 14759 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176837 14759 mem_tracker.cc:79] Creating tracker lbm-sharded_lru_cache->root 20:04:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.176944 14759 file_cache.cc:466] Constructed file cache lbm with capacity 419430 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.179354 14759 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.179411 14759 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.179500 14759 fs_report.cc:352] FS layout report 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) -------------------- 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) wal directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) metadata directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) 1 data directories: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data/data 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live blocks: 0 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes: 0 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes (after alignment): 0 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total number of LBM containers: 0 (0 full) 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for missing blocks 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for orphaned blocks 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM containers with extra space: 0 (0 repaired) 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM container extra space in bytes: 0 (0 repaired) 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total incomplete LBM containers: 0 (0 repaired) 20:04:12.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total LBM partial records: 0 (0 repaired) 20:04:12.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.233117 14759 rolling_log.cc:164] Rolled diagnostics log to new file: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/logs/kudu-master.dist-test-slave-dist-test-slave-gxc6.slave.diagnostics.20181108-200412.0.14759 20:04:12.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.233532 14759 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 20:04:12.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.234071 14759 master_main.cc:83] Starting Master server... 20:04:12.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.237895 14759 rpc_server.cc:205] RPC server started. Bound to: 127.14.104.62:45145 20:04:12.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.238199 14759 webserver.cc:175] Starting webserver on 127.14.104.62:0 20:04:12.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.238222 14759 webserver.cc:186] Document root disabled 20:04:12.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.238641 14759 webserver.cc:313] Webserver started. Bound to: http://127.14.104.62:43029/ 20:04:12.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.239073 14759 server_base.cc:607] Dumped server information to /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/data/info.pb 20:04:12.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.241441 14818 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 20:04:12.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.244410 14818 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.248934 14752 external_mini_cluster.cc:907] Started /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master as pid 14759 20:04:12.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.249579 14752 external_mini_cluster.cc:845] Running /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master 20:04:12.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master 20:04:12.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal 20:04:12.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data 20:04:12.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --block_manager=log 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=localhost 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_ca_key_size=1024 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tsk_num_rsa_bits=512 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.61:42699 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.61 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/logs 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data/info.pb 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_reuseport=true 20:04:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --master_addresses=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 with env {} 20:04:12.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.252681 14795 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.253525 14818 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.254071 14818 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.14.104.61:42699: Network error: Client connection negotiation failed: client connection to 127.14.104.61:42699: connect: Connection refused (error 111) 20:04:12.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.254097 14768 connection.cc:299] Transfer of RPC call RPC call kudu.consensus.ConsensusService.GetNodeInstance -> {remote=127.14.104.61:42699, user_credentials={real_user=slave}} aborted: Runtime error: RPC transfer destroyed before it finished sending 20:04:12.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.254122 14818 consensus_peers.cc:607] Sleeping 19 ms. before retrying to get uuid from remote peer... 20:04:12.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.273262 14818 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } attempt: 1 20:04:12.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.273911 14818 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.14.104.61:42699: Network error: Client connection negotiation failed: client connection to 127.14.104.61:42699: connect: Connection refused (error 111) 20:04:12.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.273937 14818 consensus_peers.cc:607] Sleeping 48 ms. before retrying to get uuid from remote peer... 20:04:12.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.273916 14768 connection.cc:299] Transfer of RPC call RPC call kudu.consensus.ConsensusService.GetNodeInstance -> {remote=127.14.104.61:42699, user_credentials={real_user=slave}} aborted: Runtime error: RPC transfer destroyed before it finished sending 20:04:12.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) WARNING: Logging before InitGoogleLogging() is written to STDERR 20:04:12.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.296334 14821 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 20:04:12.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.296507 14821 flags.cc:406] Enabled unsafe flag: --never_fsync=true 20:04:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.297304 14821 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 20:04:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.297327 14821 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 20:04:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.297333 14821 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 20:04:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.297339 14821 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 20:04:12.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.298806 14821 master_main.cc:73] Master server non-default flags: 20:04:12.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data 20:04:12.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal 20:04:12.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_ca_key_size=1024 20:04:12.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --master_addresses=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tsk_num_rsa_bits=512 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.61:42699 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_reuseport=true 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports=true 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data/info.pb 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.61 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync=true 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --heap_profile_path=/tmp/kudu-master.14821 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags=true 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags=true 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/logs 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr=true 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Master server version: 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) kudu 1.9.0-SNAPSHOT 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) revision 94e447869fb2e786919410e6a986beac4e6ef49d 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) build type DEBUG 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) built by ghenke at 05 Nov 2018 12:54:17 PST on ve0518.halxg.cloudera.com 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.299067 14821 mem_tracker.cc:79] Creating tracker root 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.299167 14821 mem_tracker.cc:79] Creating tracker server->root 20:04:12.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.299561 14821 mem_tracker.cc:79] Creating tracker result-tracker->server->root 20:04:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.301518 14821 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.301651 14821 mem_tracker.cc:79] Creating tracker code_cache-sharded_lru_cache->root 20:04:12.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.303303 14821 master_main.cc:80] Initializing master server... 20:04:12.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.303342 14821 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.303366 14821 mem_tracker.cc:79] Creating tracker block_cache-sharded_lru_cache->root 20:04:12.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.303721 14821 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 109960us 20:04:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.304507 14821 fs_manager.cc:263] Metadata directory not provided 20:04:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.304548 14821 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 20:04:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.304554 14821 fs_manager.cc:289] WAL root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal 20:04:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.304561 14821 fs_manager.cc:290] Metadata root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal 20:04:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.304595 14821 fs_manager.cc:291] Data roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data 20:04:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.304602 14821 fs_manager.cc:293] All roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal 20:04:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.304817 14821 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file 20:04:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.304843 14821 server_base.cc:433] Attempting to create new FS layout instead 20:04:12.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.306365 14821 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data/instance: 20:04:12.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "53546bfe72ce43e99dcc08fee1ec6b84" 20:04:12.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.306614 14821 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal/instance: 20:04:12.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "53546bfe72ce43e99dcc08fee1ec6b84" 20:04:12.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.308946 14821 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.309090 14821 pb_util.cc:291] Reading PB with version 2 starting at offset 597 20:04:12.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.311381 14821 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s 20:04:12.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.311547 14821 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.311614 14821 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.311734 14821 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.311797 14821 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.312114 14821 mem_tracker.cc:79] Creating tracker log_block_manager->server->root 20:04:12.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.312216 14821 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 20:04:12.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.312420 14821 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.312480 14821 mem_tracker.cc:79] Creating tracker lbm-sharded_lru_cache->root 20:04:12.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.312636 14821 file_cache.cc:466] Constructed file cache lbm with capacity 419430 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.315052 14821 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.315109 14821 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "53546bfe72ce43e99dcc08fee1ec6b84" 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.315227 14821 fs_report.cc:352] FS layout report 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) -------------------- 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) wal directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) metadata directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) 1 data directories: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data/data 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live blocks: 0 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes: 0 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes (after alignment): 0 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total number of LBM containers: 0 (0 full) 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for missing blocks 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for orphaned blocks 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM containers with extra space: 0 (0 repaired) 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM container extra space in bytes: 0 (0 repaired) 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total incomplete LBM containers: 0 (0 repaired) 20:04:12.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total LBM partial records: 0 (0 repaired) 20:04:12.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.322116 14818 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } attempt: 2 20:04:12.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.322741 14768 connection.cc:299] Transfer of RPC call RPC call kudu.consensus.ConsensusService.GetNodeInstance -> {remote=127.14.104.61:42699, user_credentials={real_user=slave}} aborted: Runtime error: RPC transfer destroyed before it finished sending 20:04:12.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.322770 14818 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.14.104.61:42699: Network error: Client connection negotiation failed: client connection to 127.14.104.61:42699: connect: Connection refused (error 111) 20:04:12.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.322789 14818 consensus_peers.cc:607] Sleeping 111 ms. before retrying to get uuid from remote peer... 20:04:12.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.344864 14821 rolling_log.cc:164] Rolled diagnostics log to new file: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/logs/kudu-master.dist-test-slave-dist-test-slave-gxc6.slave.diagnostics.20181108-200412.0.14821 20:04:12.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.345250 14821 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 20:04:12.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.345752 14821 master_main.cc:83] Starting Master server... 20:04:12.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.350387 14821 rpc_server.cc:205] RPC server started. Bound to: 127.14.104.61:42699 20:04:12.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.350651 14821 webserver.cc:175] Starting webserver on 127.14.104.61:0 20:04:12.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.350668 14821 webserver.cc:186] Document root disabled 20:04:12.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.350987 14821 webserver.cc:313] Webserver started. Bound to: http://127.14.104.61:33551/ 20:04:12.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.351433 14821 server_base.cc:607] Dumped server information to /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/data/info.pb 20:04:12.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.353394 14880 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.354218 14752 external_mini_cluster.cc:907] Started /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master as pid 14821 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.354593 14752 external_mini_cluster.cc:845] Running /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --block_manager=log 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=localhost 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_ca_key_size=1024 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tsk_num_rsa_bits=512 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.60:38345 20:04:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.60 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/logs 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data/info.pb 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_reuseport=true 20:04:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --master_addresses=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 with env {} 20:04:12.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.356995 14880 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.363358 14795 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.363987 14880 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.369081 14857 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.369704 14880 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.370218 14880 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.14.104.60:38345: Network error: Client connection negotiation failed: client connection to 127.14.104.60:38345: connect: Connection refused (error 111) 20:04:12.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.370244 14828 connection.cc:299] Transfer of RPC call RPC call kudu.consensus.ConsensusService.GetNodeInstance -> {remote=127.14.104.60:38345, user_credentials={real_user=slave}} aborted: Runtime error: RPC transfer destroyed before it finished sending 20:04:12.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.370273 14880 consensus_peers.cc:607] Sleeping 19 ms. before retrying to get uuid from remote peer... 20:04:12.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.389410 14880 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } attempt: 1 20:04:12.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.389961 14880 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.14.104.60:38345: Network error: Client connection negotiation failed: client connection to 127.14.104.60:38345: connect: Connection refused (error 111) 20:04:12.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.389967 14828 connection.cc:299] Transfer of RPC call RPC call kudu.consensus.ConsensusService.GetNodeInstance -> {remote=127.14.104.60:38345, user_credentials={real_user=slave}} aborted: Runtime error: RPC transfer destroyed before it finished sending 20:04:12.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.389993 14880 consensus_peers.cc:607] Sleeping 48 ms. before retrying to get uuid from remote peer... 20:04:12.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) WARNING: Logging before InitGoogleLogging() is written to STDERR 20:04:12.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.404338 14881 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 20:04:12.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.404546 14881 flags.cc:406] Enabled unsafe flag: --never_fsync=true 20:04:12.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.405412 14881 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 20:04:12.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.405444 14881 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 20:04:12.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.405455 14881 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 20:04:12.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.405467 14881 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.407004 14881 master_main.cc:73] Master server non-default flags: 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_ca_key_size=1024 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --master_addresses=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tsk_num_rsa_bits=512 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.60:38345 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_reuseport=true 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports=true 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data/info.pb 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.60 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync=true 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --heap_profile_path=/tmp/kudu-master.14881 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags=true 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags=true 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/logs 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr=true 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Master server version: 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) kudu 1.9.0-SNAPSHOT 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) revision 94e447869fb2e786919410e6a986beac4e6ef49d 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) build type DEBUG 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) built by ghenke at 05 Nov 2018 12:54:17 PST on ve0518.halxg.cloudera.com 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.407282 14881 mem_tracker.cc:79] Creating tracker root 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.407382 14881 mem_tracker.cc:79] Creating tracker server->root 20:04:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.407742 14881 mem_tracker.cc:79] Creating tracker result-tracker->server->root 20:04:12.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.409672 14881 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.409801 14881 mem_tracker.cc:79] Creating tracker code_cache-sharded_lru_cache->root 20:04:12.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.411516 14881 master_main.cc:80] Initializing master server... 20:04:12.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.411561 14881 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.411586 14881 mem_tracker.cc:79] Creating tracker block_cache-sharded_lru_cache->root 20:04:12.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.411938 14881 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 109960us 20:04:12.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.412736 14881 fs_manager.cc:263] Metadata directory not provided 20:04:12.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.412775 14881 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 20:04:12.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.412781 14881 fs_manager.cc:289] WAL root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal 20:04:12.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.412788 14881 fs_manager.cc:290] Metadata root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal 20:04:12.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.412822 14881 fs_manager.cc:291] Data roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data 20:04:12.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.412828 14881 fs_manager.cc:293] All roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal 20:04:12.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.412983 14881 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file 20:04:12.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.413007 14881 server_base.cc:433] Attempting to create new FS layout instead 20:04:12.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.414551 14881 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data/instance: 20:04:12.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "e02fd68b10f84f04a2cbc3945512c242" 20:04:12.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.414790 14881 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal/instance: 20:04:12.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "e02fd68b10f84f04a2cbc3945512c242" 20:04:12.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.417140 14881 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.417273 14881 pb_util.cc:291] Reading PB with version 2 starting at offset 597 20:04:12.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.419665 14881 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s 20:04:12.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.419800 14881 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.419853 14881 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.419936 14881 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.419975 14881 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.420289 14881 mem_tracker.cc:79] Creating tracker log_block_manager->server->root 20:04:12.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.420369 14881 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 20:04:12.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.420553 14881 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.420590 14881 mem_tracker.cc:79] Creating tracker lbm-sharded_lru_cache->root 20:04:12.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.420694 14881 file_cache.cc:466] Constructed file cache lbm with capacity 419430 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.423125 14881 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.423175 14881 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "e02fd68b10f84f04a2cbc3945512c242" 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.423290 14881 fs_report.cc:352] FS layout report 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) -------------------- 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) wal directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) metadata directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) 1 data directories: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data/data 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live blocks: 0 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes: 0 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes (after alignment): 0 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total number of LBM containers: 0 (0 full) 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for missing blocks 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for orphaned blocks 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM containers with extra space: 0 (0 repaired) 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM container extra space in bytes: 0 (0 repaired) 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total incomplete LBM containers: 0 (0 repaired) 20:04:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total LBM partial records: 0 (0 repaired) 20:04:12.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.433943 14818 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } attempt: 3 20:04:12.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.437258 14857 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.437677 14818 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.438119 14880 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } attempt: 2 20:04:12.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.438233 14766 connection.cc:299] Transfer of RPC call RPC call kudu.consensus.ConsensusService.GetNodeInstance -> {remote=127.14.104.60:38345, user_credentials={real_user=slave}} aborted: Runtime error: RPC transfer destroyed before it finished sending 20:04:12.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.438298 14818 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.14.104.60:38345: Network error: Client connection negotiation failed: client connection to 127.14.104.60:38345: connect: Connection refused (error 111) 20:04:12.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.438313 14818 consensus_peers.cc:607] Sleeping 64 ms. before retrying to get uuid from remote peer... 20:04:12.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.438607 14828 connection.cc:299] Transfer of RPC call RPC call kudu.consensus.ConsensusService.GetNodeInstance -> {remote=127.14.104.60:38345, user_credentials={real_user=slave}} aborted: Runtime error: RPC transfer destroyed before it finished sending 20:04:12.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.438673 14880 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.14.104.60:38345: Network error: Client connection negotiation failed: client connection to 127.14.104.60:38345: connect: Connection refused (error 111) 20:04:12.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.438686 14880 consensus_peers.cc:607] Sleeping 111 ms. before retrying to get uuid from remote peer... 20:04:12.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.449626 14881 rolling_log.cc:164] Rolled diagnostics log to new file: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/logs/kudu-master.dist-test-slave-dist-test-slave-gxc6.slave.diagnostics.20181108-200412.0.14881 20:04:12.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.450158 14881 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 20:04:12.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.450875 14881 master_main.cc:83] Starting Master server... 20:04:12.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.455843 14881 rpc_server.cc:205] RPC server started. Bound to: 127.14.104.60:38345 20:04:12.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.456203 14881 webserver.cc:175] Starting webserver on 127.14.104.60:0 20:04:12.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.456231 14881 webserver.cc:186] Document root disabled 20:04:12.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.456689 14881 webserver.cc:313] Webserver started. Bound to: http://127.14.104.60:42655/ 20:04:12.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.457253 14881 server_base.cc:607] Dumped server information to /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/data/info.pb 20:04:12.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.459553 14752 external_mini_cluster.cc:907] Started /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-master as pid 14881 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.460089 14752 external_mini_cluster.cc:845] Running /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --block_manager=log 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.1:0 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --local_ip_for_outbound_sockets=127.14.104.1 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.1 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tserver_master_addrs=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/logs 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data/info.pb 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 with env {} 20:04:12.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.460145 14942 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 20:04:12.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.468318 14942 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.476006 14795 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.476634 14942 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.479635 14857 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.479985 14942 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } has no permanent_uuid. Determining permanent_uuid... 20:04:12.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.487610 14919 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.488483 14942 sys_catalog.cc:316] Distributed Raft configuration: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.488682 14942 consensus_meta.cc:318] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.490284 14942 tablet_metadata.cc:505] Number of flush pins: 1 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.490321 14942 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Bootstrap starting. 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.490591 14942 tablet_bootstrap.cc:566] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Tablet Metadata: table_id: "sys.catalog.id" 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) tablet_id: "00000000000000000000000000000000" 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) last_durable_mrs_id: -1 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) table_name: "sys.catalog" 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) schema { 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 10 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "entry_type" 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: INT8 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: true 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 11 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "entry_id" 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: STRING 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: true 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 12 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "metadata" 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: STRING 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: false 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) schema_version: 0 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) tablet_data_state: TABLET_DATA_READY 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition { 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_key_start: "" 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_key_end: "" 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_schema { 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) range_schema { 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 10 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 11 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) data_dir_group { 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuids: "e75f01c261514fa09e87b0edf3afc4a1" 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.490778 14942 mem_tracker.cc:79] Creating tracker tablet-00000000000000000000000000000000->server->root 20:04:12.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.490809 14942 mem_tracker.cc:79] Creating tracker DeltaMemStores->tablet-00000000000000000000000000000000->server->root 20:04:12.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.493077 14942 mem_tracker.cc:79] Creating tracker MemRowSet-0->tablet-00000000000000000000000000000000->server->root 20:04:12.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.493412 14942 log_util.cc:863] Ignoring hidden file: .. 20:04:12.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.493446 14942 log_util.cc:863] Ignoring hidden file: . 20:04:12.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.493460 14942 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: No blocks or log segments found. Creating new log. 20:04:12.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.493952 14942 log_reader.cc:125] Reading wal from path:/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal/wals/00000000000000000000000000000000 20:04:12.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.493988 14942 log_reader.cc:131] Parsing segments from path: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal/wals/00000000000000000000000000000000 20:04:12.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.494091 14942 log.cc:525] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Log is configured to *not* fsync() on all Append() calls 20:04:12.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.495429 14947 log.cc:1153] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Created next WAL segment, placeholder path: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-2/wal/wals/00000000000000000000000000000000/.kudutmp.newsegmentM6594g 20:04:12.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.495934 14942 log.cc:271] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Starting log append thread 20:04:12.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.496045 14942 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: No bootstrap required, opened a new log 20:04:12.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.496405 14942 mem_tracker.cc:79] Creating tracker txn_tracker->tablet-00000000000000000000000000000000->server->root 20:04:12.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.496737 14942 mem_tracker.cc:79] Creating tracker log_cache->root 20:04:12.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.496781 14942 mem_tracker.cc:79] Creating tracker log_cache:e02fd68b10f84f04a2cbc3945512c242:00000000000000000000000000000000->log_cache->root 20:04:12.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.497952 14942 consensus_meta.cc:318] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.498023 14942 raft_consensus.cc:339] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.498087 14942 raft_consensus.cc:365] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 20:04:12.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.498131 14942 raft_consensus.cc:704] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02fd68b10f84f04a2cbc3945512c242, State: Initialized, Role: FOLLOWER 20:04:12.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.498179 14942 consensus_meta.cc:318] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.498550 14942 consensus_queue.cc:229] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.498785 14942 sys_catalog.cc:337] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.498817 14942 sys_catalog.cc:340] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.498986 14942 maintenance_manager.cc:211] P e02fd68b10f84f04a2cbc3945512c242: Registered FlushMRSOp(00000000000000000000000000000000) 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499147 14942 maintenance_manager.cc:211] P e02fd68b10f84f04a2cbc3945512c242: Registered FlushDeltaMemStoresOp(00000000000000000000000000000000) 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499249 14942 maintenance_manager.cc:211] P e02fd68b10f84f04a2cbc3945512c242: Registered LogGCOp(00000000000000000000000000000000) 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499322 14942 maintenance_manager.cc:211] P e02fd68b10f84f04a2cbc3945512c242: Registered CompactRowSetsOp(00000000000000000000000000000000) 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499358 14942 maintenance_manager.cc:211] P e02fd68b10f84f04a2cbc3945512c242: Registered MinorDeltaCompactionOp(00000000000000000000000000000000) 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499382 14948 sys_catalog.cc:337] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499424 14948 sys_catalog.cc:340] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499385 14942 maintenance_manager.cc:211] P e02fd68b10f84f04a2cbc3945512c242: Registered MajorDeltaCompactionOp(00000000000000000000000000000000) 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499461 14942 maintenance_manager.cc:211] P e02fd68b10f84f04a2cbc3945512c242: Registered UndoDeltaBlockGCOp(00000000000000000000000000000000) 20:04:12.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.499891 14942 sys_catalog.cc:424] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: configured and running, proceeding with master startup. 20:04:12.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.500253 14881 master_main.cc:86] Master server successfully started. 20:04:12.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.502465 14818 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } attempt: 1 20:04:12.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.503033 14949 catalog_manager.cc:1091] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: acquiring CA information for follower catalog manager: Not found: root CA entry not found 20:04:12.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.503054 14949 catalog_manager.cc:584] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 20:04:12.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.506090 14919 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.506604 14818 sys_catalog.cc:316] Distributed Raft configuration: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.506800 14818 consensus_meta.cc:318] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.508066 14818 tablet_metadata.cc:505] Number of flush pins: 1 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.508108 14818 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Bootstrap starting. 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.508291 14818 tablet_bootstrap.cc:566] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Tablet Metadata: table_id: "sys.catalog.id" 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) tablet_id: "00000000000000000000000000000000" 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) last_durable_mrs_id: -1 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) table_name: "sys.catalog" 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) schema { 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 10 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "entry_type" 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: INT8 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: true 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 11 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "entry_id" 20:04:12.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: STRING 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: true 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 12 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "metadata" 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: STRING 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: false 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) schema_version: 0 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) tablet_data_state: TABLET_DATA_READY 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition { 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_key_start: "" 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_key_end: "" 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_schema { 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) range_schema { 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 10 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 11 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) data_dir_group { 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuids: "e335a3246cb14e6bafb7f8ac0aa6393b" 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.508483 14818 mem_tracker.cc:79] Creating tracker tablet-00000000000000000000000000000000->server->root 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.508520 14818 mem_tracker.cc:79] Creating tracker DeltaMemStores->tablet-00000000000000000000000000000000->server->root 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.509733 14818 mem_tracker.cc:79] Creating tracker MemRowSet-0->tablet-00000000000000000000000000000000->server->root 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.510103 14818 log_util.cc:863] Ignoring hidden file: .. 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.510131 14818 log_util.cc:863] Ignoring hidden file: . 20:04:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.510144 14818 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: No blocks or log segments found. Creating new log. 20:04:12.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.511097 14818 log_reader.cc:125] Reading wal from path:/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal/wals/00000000000000000000000000000000 20:04:12.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.511135 14818 log_reader.cc:131] Parsing segments from path: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal/wals/00000000000000000000000000000000 20:04:12.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.511258 14818 log.cc:525] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Log is configured to *not* fsync() on all Append() calls 20:04:12.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.514649 14951 log.cc:1153] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Created next WAL segment, placeholder path: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-0/wal/wals/00000000000000000000000000000000/.kudutmp.newsegmentc0G6gY 20:04:12.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.515424 14818 log.cc:271] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Starting log append thread 20:04:12.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.515558 14818 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: No bootstrap required, opened a new log 20:04:12.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.515930 14818 mem_tracker.cc:79] Creating tracker txn_tracker->tablet-00000000000000000000000000000000->server->root 20:04:12.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.516257 14818 mem_tracker.cc:79] Creating tracker log_cache->root 20:04:12.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.516345 14818 mem_tracker.cc:79] Creating tracker log_cache:9bc86b6e11024fdb8efd341f6b1beb9a:00000000000000000000000000000000->log_cache->root 20:04:12.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.517273 14818 consensus_meta.cc:318] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.517313 14818 raft_consensus.cc:339] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.517344 14818 raft_consensus.cc:365] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.517365 14818 raft_consensus.cc:704] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc86b6e11024fdb8efd341f6b1beb9a, State: Initialized, Role: FOLLOWER 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.517391 14818 consensus_meta.cc:318] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.517696 14818 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.517876 14818 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.517894 14818 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518013 14818 maintenance_manager.cc:211] P 9bc86b6e11024fdb8efd341f6b1beb9a: Registered FlushMRSOp(00000000000000000000000000000000) 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518110 14818 maintenance_manager.cc:211] P 9bc86b6e11024fdb8efd341f6b1beb9a: Registered FlushDeltaMemStoresOp(00000000000000000000000000000000) 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518153 14818 maintenance_manager.cc:211] P 9bc86b6e11024fdb8efd341f6b1beb9a: Registered LogGCOp(00000000000000000000000000000000) 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518199 14818 maintenance_manager.cc:211] P 9bc86b6e11024fdb8efd341f6b1beb9a: Registered CompactRowSetsOp(00000000000000000000000000000000) 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518213 14818 maintenance_manager.cc:211] P 9bc86b6e11024fdb8efd341f6b1beb9a: Registered MinorDeltaCompactionOp(00000000000000000000000000000000) 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518239 14818 maintenance_manager.cc:211] P 9bc86b6e11024fdb8efd341f6b1beb9a: Registered MajorDeltaCompactionOp(00000000000000000000000000000000) 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518249 14818 maintenance_manager.cc:211] P 9bc86b6e11024fdb8efd341f6b1beb9a: Registered UndoDeltaBlockGCOp(00000000000000000000000000000000) 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518587 14818 sys_catalog.cc:424] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: configured and running, proceeding with master startup. 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.518879 14759 master_main.cc:86] Master server successfully started. 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.519376 14952 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.519412 14952 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.521584 14953 catalog_manager.cc:1091] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: acquiring CA information for follower catalog manager: Not found: root CA entry not found 20:04:12.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.521620 14953 catalog_manager.cc:584] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 20:04:12.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) WARNING: Logging before InitGoogleLogging() is written to STDERR 20:04:12.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.525321 14943 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 20:04:12.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.525538 14943 flags.cc:406] Enabled unsafe flag: --never_fsync=true 20:04:12.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.526643 14943 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 20:04:12.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.526723 14943 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.104.1 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.528650 14943 tablet_server_main.cc:78] Tablet server non-default flags: 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.1:0 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports=true 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data/info.pb 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.1 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tserver_master_addrs=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync=true 20:04:12.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --heap_profile_path=/tmp/kudu-tserver.14943 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags=true 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags=true 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --local_ip_for_outbound_sockets=127.14.104.1 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/logs 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr=true 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Tablet server version: 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) kudu 1.9.0-SNAPSHOT 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) revision 94e447869fb2e786919410e6a986beac4e6ef49d 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) build type DEBUG 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) built by ghenke at 05 Nov 2018 12:54:17 PST on ve0518.halxg.cloudera.com 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.529001 14943 mem_tracker.cc:79] Creating tracker root 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.529137 14943 mem_tracker.cc:79] Creating tracker server->root 20:04:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.529567 14943 mem_tracker.cc:79] Creating tracker result-tracker->server->root 20:04:12.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.532402 14943 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.532591 14943 mem_tracker.cc:79] Creating tracker code_cache-sharded_lru_cache->root 20:04:12.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.535373 14943 tablet_server_main.cc:85] Initializing tablet server... 20:04:12.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.535425 14943 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.535459 14943 mem_tracker.cc:79] Creating tracker block_cache-sharded_lru_cache->root 20:04:12.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.536423 14943 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 109960us 20:04:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.536975 14943 fs_manager.cc:263] Metadata directory not provided 20:04:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.537034 14943 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 20:04:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.537045 14943 fs_manager.cc:289] WAL root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal 20:04:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.537056 14943 fs_manager.cc:290] Metadata root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal 20:04:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.537117 14943 fs_manager.cc:291] Data roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data 20:04:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.537128 14943 fs_manager.cc:293] All roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal 20:04:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.537322 14943 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file 20:04:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.537353 14943 server_base.cc:433] Attempting to create new FS layout instead 20:04:12.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.539377 14943 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data/instance: 20:04:12.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "04755666e2fa429cbf40a5b93e2db34f" 20:04:12.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.539728 14943 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal/instance: 20:04:12.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "04755666e2fa429cbf40a5b93e2db34f" 20:04:12.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.543181 14943 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.543404 14943 pb_util.cc:291] Reading PB with version 2 starting at offset 597 20:04:12.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.548560 14943 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.002s sys 0.004s 20:04:12.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.548693 14943 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.548755 14943 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.548856 14943 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.548907 14943 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.549265 14943 mem_tracker.cc:79] Creating tracker log_block_manager->server->root 20:04:12.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.549358 14943 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 20:04:12.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.549563 14943 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.549610 14943 mem_tracker.cc:79] Creating tracker lbm-sharded_lru_cache->root 20:04:12.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.549731 14943 file_cache.cc:466] Constructed file cache lbm with capacity 419430 20:04:12.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.549825 14880 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } attempt: 3 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.553071 14943 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.553133 14943 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "04755666e2fa429cbf40a5b93e2db34f" 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.553232 14943 fs_report.cc:352] FS layout report 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) -------------------- 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) wal directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) metadata directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/wal 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) 1 data directories: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data/data 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live blocks: 0 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes: 0 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes (after alignment): 0 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total number of LBM containers: 0 (0 full) 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for missing blocks 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for orphaned blocks 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM containers with extra space: 0 (0 repaired) 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM container extra space in bytes: 0 (0 repaired) 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total incomplete LBM containers: 0 (0 repaired) 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total LBM partial records: 0 (0 repaired) 20:04:12.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.553910 14919 tablet_service.cc:1110] Received Get Node Instance RPC: 20:04:12.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.554515 14880 sys_catalog.cc:316] Distributed Raft configuration: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.554728 14880 consensus_meta.cc:318] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.556457 14880 tablet_metadata.cc:505] Number of flush pins: 1 20:04:12.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.556514 14880 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Bootstrap starting. 20:04:12.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.556721 14880 tablet_bootstrap.cc:566] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Tablet Metadata: table_id: "sys.catalog.id" 20:04:12.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) tablet_id: "00000000000000000000000000000000" 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) last_durable_mrs_id: -1 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) table_name: "sys.catalog" 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) schema { 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 10 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "entry_type" 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: INT8 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: true 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 11 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "entry_id" 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: STRING 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: true 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 12 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) name: "metadata" 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) type: STRING 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_key: false 20:04:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) is_nullable: false 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) encoding: AUTO_ENCODING 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) compression: DEFAULT_COMPRESSION 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) cfile_block_size: 0 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) schema_version: 0 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) tablet_data_state: TABLET_DATA_READY 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition { 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_key_start: "" 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_key_end: "" 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) partition_schema { 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) range_schema { 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 10 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) columns { 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) id: 11 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) data_dir_group { 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuids: "6a05b924d1774fc9b67343393ad4ccb3" 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) } 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.556906 14880 mem_tracker.cc:79] Creating tracker tablet-00000000000000000000000000000000->server->root 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.556938 14880 mem_tracker.cc:79] Creating tracker DeltaMemStores->tablet-00000000000000000000000000000000->server->root 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.558424 14880 mem_tracker.cc:79] Creating tracker MemRowSet-0->tablet-00000000000000000000000000000000->server->root 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.558790 14880 log_util.cc:863] Ignoring hidden file: .. 20:04:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.558818 14880 log_util.cc:863] Ignoring hidden file: . 20:04:12.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.558830 14880 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: No blocks or log segments found. Creating new log. 20:04:12.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.559425 14880 log_reader.cc:125] Reading wal from path:/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal/wals/00000000000000000000000000000000 20:04:12.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.559453 14880 log_reader.cc:131] Parsing segments from path: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal/wals/00000000000000000000000000000000 20:04:12.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.559551 14880 log.cc:525] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Log is configured to *not* fsync() on all Append() calls 20:04:12.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.563413 14961 log.cc:1153] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Created next WAL segment, placeholder path: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/master-1/wal/wals/00000000000000000000000000000000/.kudutmp.newsegmentucz8Da 20:04:12.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.564057 14880 log.cc:271] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Starting log append thread 20:04:12.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.564157 14880 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: No bootstrap required, opened a new log 20:04:12.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.564544 14880 mem_tracker.cc:79] Creating tracker txn_tracker->tablet-00000000000000000000000000000000->server->root 20:04:12.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.564889 14880 mem_tracker.cc:79] Creating tracker log_cache->root 20:04:12.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.564934 14880 mem_tracker.cc:79] Creating tracker log_cache:53546bfe72ce43e99dcc08fee1ec6b84:00000000000000000000000000000000->log_cache->root 20:04:12.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.566134 14880 consensus_meta.cc:318] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.566205 14880 raft_consensus.cc:339] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.566249 14880 raft_consensus.cc:365] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 20:04:12.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.566287 14880 raft_consensus.cc:704] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53546bfe72ce43e99dcc08fee1ec6b84, State: Initialized, Role: FOLLOWER 20:04:12.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.566329 14880 consensus_meta.cc:318] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Updating active role to FOLLOWER. Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.566730 14880 consensus_queue.cc:229] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.566978 14880 sys_catalog.cc:337] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567011 14880 sys_catalog.cc:340] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567181 14880 maintenance_manager.cc:211] P 53546bfe72ce43e99dcc08fee1ec6b84: Registered FlushMRSOp(00000000000000000000000000000000) 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567353 14966 sys_catalog.cc:337] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567365 14880 maintenance_manager.cc:211] P 53546bfe72ce43e99dcc08fee1ec6b84: Registered FlushDeltaMemStoresOp(00000000000000000000000000000000) 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567387 14966 sys_catalog.cc:340] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567446 14880 maintenance_manager.cc:211] P 53546bfe72ce43e99dcc08fee1ec6b84: Registered LogGCOp(00000000000000000000000000000000) 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567518 14880 maintenance_manager.cc:211] P 53546bfe72ce43e99dcc08fee1ec6b84: Registered CompactRowSetsOp(00000000000000000000000000000000) 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567540 14880 maintenance_manager.cc:211] P 53546bfe72ce43e99dcc08fee1ec6b84: Registered MinorDeltaCompactionOp(00000000000000000000000000000000) 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567560 14880 maintenance_manager.cc:211] P 53546bfe72ce43e99dcc08fee1ec6b84: Registered MajorDeltaCompactionOp(00000000000000000000000000000000) 20:04:12.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567576 14880 maintenance_manager.cc:211] P 53546bfe72ce43e99dcc08fee1ec6b84: Registered UndoDeltaBlockGCOp(00000000000000000000000000000000) 20:04:12.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.567987 14880 sys_catalog.cc:424] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: configured and running, proceeding with master startup. 20:04:12.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.568406 14821 master_main.cc:86] Master server successfully started. 20:04:12.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.572933 14967 catalog_manager.cc:1091] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: acquiring CA information for follower catalog manager: Not found: root CA entry not found 20:04:12.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.572969 14967 catalog_manager.cc:584] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 20:04:12.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.590039 14943 rolling_log.cc:164] Rolled diagnostics log to new file: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/logs/kudu-tserver.dist-test-slave-dist-test-slave-gxc6.slave.diagnostics.20181108-200412.0.14943 20:04:12.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.590440 14943 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 20:04:12.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.595331 14943 fs_manager.cc:628] Ignoring hidden file in tablet metadata dir: .. 20:04:12.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.595355 14943 fs_manager.cc:628] Ignoring hidden file in tablet metadata dir: . 20:04:12.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.595489 14943 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 20:04:12.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.595603 14943 tablet_server_main.cc:90] Starting tablet server... 20:04:12.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.596134 14836 tablet.cc:1744] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Can't schedule compaction. Clean time has not been advanced past its initial value. 20:04:12.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.602314 14943 rpc_server.cc:205] RPC server started. Bound to: 127.14.104.1:41675 20:04:12.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.602639 14943 webserver.cc:175] Starting webserver on 127.14.104.1:0 20:04:12.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.602663 14943 webserver.cc:186] Document root disabled 20:04:12.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.603121 14943 webserver.cc:313] Webserver started. Bound to: http://127.14.104.1:36801/ 20:04:12.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.603678 14943 server_base.cc:607] Dumped server information to /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-0/data/info.pb 20:04:12.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.604770 14943 tablet_server_main.cc:93] Tablet server successfully started. 20:04:12.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.604441 15077 heartbeater.cc:549] Heartbeat thread (master 127.14.104.62:45145) starting 20:04:12.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.604921 15077 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.604846 15079 heartbeater.cc:549] Heartbeat thread (master 127.14.104.60:38345) starting 20:04:12.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.604959 15079 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.607136 14752 external_mini_cluster.cc:907] Started /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver as pid 14943 20:04:12.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.607820 14752 external_mini_cluster.cc:845] Running /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --block_manager=log 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.2:0 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --local_ip_for_outbound_sockets=127.14.104.2 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.2 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tserver_master_addrs=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/logs 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data/info.pb 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 with env {} 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.604812 15078 heartbeater.cc:549] Heartbeat thread (master 127.14.104.61:42699) starting 20:04:12.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.608094 15078 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.619258 15078 heartbeater.cc:346] Connected to a master server at 127.14.104.61:42699 20:04:12.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.619487 15078 heartbeater.cc:426] Registering TS with master... 20:04:12.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.619909 15078 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.620082 15078 heartbeater.cc:475] Master 127.14.104.61:42699 requested a full tablet report, sending... 20:04:12.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.621357 14847 ts_manager.cc:98] Registered new tserver with Master: 04755666e2fa429cbf40a5b93e2db34f (127.14.104.1:41675) 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.623708 15077 heartbeater.cc:346] Connected to a master server at 127.14.104.62:45145 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.623744 15077 heartbeater.cc:426] Registering TS with master... 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.623759 15079 heartbeater.cc:346] Connected to a master server at 127.14.104.60:38345 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.623796 15079 heartbeater.cc:426] Registering TS with master... 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.623850 15077 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.623873 15077 heartbeater.cc:475] Master 127.14.104.62:45145 requested a full tablet report, sending... 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.623919 15079 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.623944 15079 heartbeater.cc:475] Master 127.14.104.60:38345 requested a full tablet report, sending... 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.624836 14777 ts_manager.cc:98] Registered new tserver with Master: 04755666e2fa429cbf40a5b93e2db34f (127.14.104.1:41675) 20:04:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.624882 14901 ts_manager.cc:98] Registered new tserver with Master: 04755666e2fa429cbf40a5b93e2db34f (127.14.104.1:41675) 20:04:12.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) WARNING: Logging before InitGoogleLogging() is written to STDERR 20:04:12.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.698913 15084 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 20:04:12.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.699149 15084 flags.cc:406] Enabled unsafe flag: --never_fsync=true 20:04:12.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.700260 15084 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 20:04:12.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.700451 15084 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.104.2 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.701870 14898 tablet.cc:1744] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Can't schedule compaction. Clean time has not been advanced past its initial value. 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.702437 15084 tablet_server_main.cc:78] Tablet server non-default flags: 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.2:0 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports=true 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data/info.pb 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.2 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tserver_master_addrs=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync=true 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --heap_profile_path=/tmp/kudu-tserver.15084 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags=true 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags=true 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --local_ip_for_outbound_sockets=127.14.104.2 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/logs 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr=true 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Tablet server version: 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) kudu 1.9.0-SNAPSHOT 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) revision 94e447869fb2e786919410e6a986beac4e6ef49d 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) build type DEBUG 20:04:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) built by ghenke at 05 Nov 2018 12:54:17 PST on ve0518.halxg.cloudera.com 20:04:12.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.703955 15084 mem_tracker.cc:79] Creating tracker root 20:04:12.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.704118 15084 mem_tracker.cc:79] Creating tracker server->root 20:04:12.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.704579 15084 mem_tracker.cc:79] Creating tracker result-tracker->server->root 20:04:12.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.707433 15084 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.707631 15084 mem_tracker.cc:79] Creating tracker code_cache-sharded_lru_cache->root 20:04:12.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.710393 15084 tablet_server_main.cc:85] Initializing tablet server... 20:04:12.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.710446 15084 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.710496 15084 mem_tracker.cc:79] Creating tracker block_cache-sharded_lru_cache->root 20:04:12.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.711560 15084 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 109960us 20:04:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.712158 15084 fs_manager.cc:263] Metadata directory not provided 20:04:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.712226 15084 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 20:04:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.712236 15084 fs_manager.cc:289] WAL root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal 20:04:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.712249 15084 fs_manager.cc:290] Metadata root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal 20:04:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.712308 15084 fs_manager.cc:291] Data roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data 20:04:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.712321 15084 fs_manager.cc:293] All roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal 20:04:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.712568 15084 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file 20:04:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.712680 15084 server_base.cc:433] Attempting to create new FS layout instead 20:04:12.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.714888 15084 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data/instance: 20:04:12.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "6728aa9ff2594a2cbddb065e49170a8f" 20:04:12.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.715239 15084 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal/instance: 20:04:12.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "6728aa9ff2594a2cbddb065e49170a8f" 20:04:12.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.718870 15084 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.719074 15084 pb_util.cc:291] Reading PB with version 2 starting at offset 597 20:04:12.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.722066 15084 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s 20:04:12.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.722218 15084 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.722286 15084 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.722409 15084 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.722488 15084 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.722834 15084 mem_tracker.cc:79] Creating tracker log_block_manager->server->root 20:04:12.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.722934 15084 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 20:04:12.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.723109 15084 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.723181 15084 mem_tracker.cc:79] Creating tracker lbm-sharded_lru_cache->root 20:04:12.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.723352 15084 file_cache.cc:466] Constructed file cache lbm with capacity 419430 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.726032 15084 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.726095 15084 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "6728aa9ff2594a2cbddb065e49170a8f" 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.726188 15084 fs_report.cc:352] FS layout report 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) -------------------- 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) wal directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) metadata directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/wal 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) 1 data directories: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data/data 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live blocks: 0 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes: 0 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes (after alignment): 0 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total number of LBM containers: 0 (0 full) 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for missing blocks 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for orphaned blocks 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM containers with extra space: 0 (0 repaired) 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM container extra space in bytes: 0 (0 repaired) 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total incomplete LBM containers: 0 (0 repaired) 20:04:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total LBM partial records: 0 (0 repaired) 20:04:12.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.734671 14774 tablet.cc:1744] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Can't schedule compaction. Clean time has not been advanced past its initial value. 20:04:12.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.766713 15084 rolling_log.cc:164] Rolled diagnostics log to new file: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/logs/kudu-tserver.dist-test-slave-dist-test-slave-gxc6.slave.diagnostics.20181108-200412.0.15084 20:04:12.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.767271 15084 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 20:04:12.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.768157 15084 fs_manager.cc:628] Ignoring hidden file in tablet metadata dir: .. 20:04:12.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.768179 15084 fs_manager.cc:628] Ignoring hidden file in tablet metadata dir: . 20:04:12.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.768297 15084 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 20:04:12.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.768401 15084 tablet_server_main.cc:90] Starting tablet server... 20:04:12.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.778216 15084 rpc_server.cc:205] RPC server started. Bound to: 127.14.104.2:34607 20:04:12.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.778504 15084 webserver.cc:175] Starting webserver on 127.14.104.2:0 20:04:12.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.778527 15084 webserver.cc:186] Document root disabled 20:04:12.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.778911 15084 webserver.cc:313] Webserver started. Bound to: http://127.14.104.2:44045/ 20:04:12.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.779392 15084 server_base.cc:607] Dumped server information to /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-1/data/info.pb 20:04:12.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.779899 15084 tablet_server_main.cc:93] Tablet server successfully started. 20:04:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.782411 15203 heartbeater.cc:549] Heartbeat thread (master 127.14.104.62:45145) starting 20:04:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.782464 15203 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.782908 15204 heartbeater.cc:549] Heartbeat thread (master 127.14.104.61:42699) starting 20:04:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.782933 15204 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.783021 15205 heartbeater.cc:549] Heartbeat thread (master 127.14.104.60:38345) starting 20:04:12.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.783052 15205 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.788967 14752 external_mini_cluster.cc:907] Started /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver as pid 15084 20:04:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.789515 14752 external_mini_cluster.cc:845] Running /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver 20:04:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver 20:04:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal 20:04:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data 20:04:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --block_manager=log 20:04:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.3:0 20:04:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --local_ip_for_outbound_sockets=127.14.104.3 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.3 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tserver_master_addrs=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/logs 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data/info.pb 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 with env {} 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.790419 15205 heartbeater.cc:346] Connected to a master server at 127.14.104.60:38345 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.790537 15205 heartbeater.cc:426] Registering TS with master... 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.790802 15205 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.790834 15205 heartbeater.cc:475] Master 127.14.104.60:38345 requested a full tablet report, sending... 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.791038 15203 heartbeater.cc:346] Connected to a master server at 127.14.104.62:45145 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.791067 15203 heartbeater.cc:426] Registering TS with master... 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.791167 15203 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.791193 15203 heartbeater.cc:475] Master 127.14.104.62:45145 requested a full tablet report, sending... 20:04:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.791749 14901 ts_manager.cc:98] Registered new tserver with Master: 6728aa9ff2594a2cbddb065e49170a8f (127.14.104.2:34607) 20:04:12.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.791909 14777 ts_manager.cc:98] Registered new tserver with Master: 6728aa9ff2594a2cbddb065e49170a8f (127.14.104.2:34607) 20:04:12.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.793246 15204 heartbeater.cc:346] Connected to a master server at 127.14.104.61:42699 20:04:12.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.793303 15204 heartbeater.cc:426] Registering TS with master... 20:04:12.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.793414 15204 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.793464 15204 heartbeater.cc:475] Master 127.14.104.61:42699 requested a full tablet report, sending... 20:04:12.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.794111 14847 ts_manager.cc:98] Registered new tserver with Master: 6728aa9ff2594a2cbddb065e49170a8f (127.14.104.2:34607) 20:04:12.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) WARNING: Logging before InitGoogleLogging() is written to STDERR 20:04:12.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.830296 15210 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 20:04:12.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.830492 15210 flags.cc:406] Enabled unsafe flag: --never_fsync=true 20:04:12.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.831359 15210 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 20:04:12.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) W1108 20:04:12.831447 15210 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.104.3 20:04:12.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.832801 15210 tablet_server_main.cc:78] Tablet server non-default flags: 20:04:12.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_data_dirs=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data 20:04:12.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --fs_wal_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal 20:04:12.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --ipki_server_key_size=1024 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_bind_addresses=127.14.104.3:0 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --rpc_server_allow_ephemeral_ports=true 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --metrics_log_interval_ms=1000 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_format=pb 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --server_dump_info_path=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data/info.pb 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_interface=127.14.104.3 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --webserver_port=0 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --tserver_master_addrs=127.14.104.62:45145,127.14.104.61:42699,127.14.104.60:38345 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --never_fsync=true 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --heap_profile_path=/tmp/kudu-tserver.15210 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --redact=none 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_experimental_flags=true 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --unlock_unsafe_flags=true 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --enable_minidumps=false 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --local_ip_for_outbound_sockets=127.14.104.3 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --log_dir=/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/logs 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logbuflevel=-1 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --logtostderr=true 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) --v=1 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Tablet server version: 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) kudu 1.9.0-SNAPSHOT 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) revision 94e447869fb2e786919410e6a986beac4e6ef49d 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) build type DEBUG 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) built by ghenke at 05 Nov 2018 12:54:17 PST on ve0518.halxg.cloudera.com 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.833143 15210 mem_tracker.cc:79] Creating tracker root 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.833266 15210 mem_tracker.cc:79] Creating tracker server->root 20:04:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.833688 15210 mem_tracker.cc:79] Creating tracker result-tracker->server->root 20:04:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.835765 15210 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.835921 15210 mem_tracker.cc:79] Creating tracker code_cache-sharded_lru_cache->root 20:04:12.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.837744 15210 tablet_server_main.cc:85] Initializing tablet server... 20:04:12.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.837792 15210 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.837826 15210 mem_tracker.cc:79] Creating tracker block_cache-sharded_lru_cache->root 20:04:12.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.838573 15210 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 109960us 20:04:12.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.838991 15210 fs_manager.cc:263] Metadata directory not provided 20:04:12.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.839046 15210 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 20:04:12.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.839058 15210 fs_manager.cc:289] WAL root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal 20:04:12.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.839071 15210 fs_manager.cc:290] Metadata root: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal 20:04:12.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.839130 15210 fs_manager.cc:291] Data roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data 20:04:12.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.839144 15210 fs_manager.cc:293] All roots: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal 20:04:12.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.839368 15210 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file 20:04:12.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.839395 15210 server_base.cc:433] Attempting to create new FS layout instead 20:04:12.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.840906 15210 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data/instance: 20:04:12.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "623d14a641dc464897d0f410e82a07b5" 20:04:12.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.841233 15210 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal/instance: 20:04:12.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "623d14a641dc464897d0f410e82a07b5" 20:04:12.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.843667 15210 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.843834 15210 pb_util.cc:291] Reading PB with version 2 starting at offset 597 20:04:12.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.845856 15210 fs_manager.cc:503] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.845980 15210 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.846053 15210 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.846163 15210 pb_util.cc:291] Reading PB with version 2 starting at offset 16 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.846216 15210 pb_util.cc:291] Reading PB with version 2 starting at offset 593 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.846529 15210 mem_tracker.cc:79] Creating tracker log_block_manager->server->root 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.846621 15210 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.846786 15210 cache.cc:441] Will use 4 shards for LRU cache. 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.846829 15210 mem_tracker.cc:79] Creating tracker lbm-sharded_lru_cache->root 20:04:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.846951 15210 file_cache.cc:466] Constructed file cache lbm with capacity 419430 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.849231 15210 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.849282 15210 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data,/tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) uuid: "623d14a641dc464897d0f410e82a07b5" 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) format_stamp: "Formatted at 2018-11-08 20:04:12 on dist-test-slave-dist-test-slave-gxc6" 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.849371 15210 fs_report.cc:352] FS layout report 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) -------------------- 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) wal directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) metadata directory: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/wal 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) 1 data directories: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data/data 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live blocks: 0 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes: 0 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total live bytes (after alignment): 0 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total number of LBM containers: 0 (0 full) 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for missing blocks 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Did not check for orphaned blocks 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM containers with extra space: 0 (0 repaired) 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total full LBM container extra space in bytes: 0 (0 repaired) 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total incomplete LBM containers: 0 (0 repaired) 20:04:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) Total LBM partial records: 0 (0 repaired) 20:04:12.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.863981 15210 rolling_log.cc:164] Rolled diagnostics log to new file: /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/logs/kudu-tserver.dist-test-slave-dist-test-slave-gxc6.slave.diagnostics.20181108-200412.0.15210 20:04:12.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.864379 15210 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 20:04:12.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.865188 15210 fs_manager.cc:628] Ignoring hidden file in tablet metadata dir: .. 20:04:12.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.865211 15210 fs_manager.cc:628] Ignoring hidden file in tablet metadata dir: . 20:04:12.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.865319 15210 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 20:04:12.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.865425 15210 tablet_server_main.cc:90] Starting tablet server... 20:04:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.870354 15210 rpc_server.cc:205] RPC server started. Bound to: 127.14.104.3:34243 20:04:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.870620 15210 webserver.cc:175] Starting webserver on 127.14.104.3:0 20:04:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.870640 15210 webserver.cc:186] Document root disabled 20:04:12.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.871002 15210 webserver.cc:313] Webserver started. Bound to: http://127.14.104.3:37623/ 20:04:12.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.871976 15210 server_base.cc:607] Dumped server information to /tmp/dist-test-taskRgtgLL/test-tmp/mini-kudu-cluster9119273347380302965/ts-2/data/info.pb 20:04:12.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.872198 15329 heartbeater.cc:549] Heartbeat thread (master 127.14.104.62:45145) starting 20:04:12.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.872548 15210 tablet_server_main.cc:93] Tablet server successfully started. 20:04:12.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.872247 15329 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.872287 15330 heartbeater.cc:549] Heartbeat thread (master 127.14.104.61:42699) starting 20:04:12.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.872604 15330 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.872318 15331 heartbeater.cc:549] Heartbeat thread (master 127.14.104.60:38345) starting 20:04:12.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.873062 15331 heartbeater.cc:415] No valid master proxy. Connecting... 20:04:12.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.873303 14752 external_mini_cluster.cc:907] Started /tmp/dist-test-taskRgtgLL/build/debug/bin/kudu-tserver as pid 15210 20:04:12.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.881587 15330 heartbeater.cc:346] Connected to a master server at 127.14.104.61:42699 20:04:12.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.881628 15329 heartbeater.cc:346] Connected to a master server at 127.14.104.62:45145 20:04:12.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.881685 15329 heartbeater.cc:426] Registering TS with master... 20:04:12.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.881989 15329 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.882040 15329 heartbeater.cc:475] Master 127.14.104.62:45145 requested a full tablet report, sending... 20:04:12.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.881685 15330 heartbeater.cc:426] Registering TS with master... 20:04:12.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.882349 15330 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.882376 15330 heartbeater.cc:475] Master 127.14.104.61:42699 requested a full tablet report, sending... 20:04:12.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.883605 15331 heartbeater.cc:346] Connected to a master server at 127.14.104.60:38345 20:04:12.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.883646 15331 heartbeater.cc:426] Registering TS with master... 20:04:12.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.883664 14777 ts_manager.cc:98] Registered new tserver with Master: 623d14a641dc464897d0f410e82a07b5 (127.14.104.3:34243) 20:04:12.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.883739 14847 ts_manager.cc:98] Registered new tserver with Master: 623d14a641dc464897d0f410e82a07b5 (127.14.104.3:34243) 20:04:12.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.883761 15331 heartbeater.cc:459] Sending a CSR to the master in the next heartbeat 20:04:12.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.883790 15331 heartbeater.cc:475] Master 127.14.104.60:38345 requested a full tablet report, sending... 20:04:12.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.884740 14901 ts_manager.cc:98] Registered new tserver with Master: 623d14a641dc464897d0f410e82a07b5 (127.14.104.3:34243) 20:04:12.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.891948 14752 external_mini_cluster.cc:535] 3 TS(s) registered with all masters 20:04:12.893 [INFO - main] (KuduTestHarness.java:139) Creating a new Kudu client... 20:04:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.896867 14952 raft_consensus.cc:471] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 20:04:12.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.896965 14952 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Snoozing failure detection for 1.830s (starting election) 20:04:12.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.897078 14952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.897428 14952 leader_election.cc:187] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 pre-election: Running leader election. 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.897879 14952 leader_election.cc:251] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 53546bfe72ce43e99dcc08fee1ec6b84 (127.14.104.61:42699) 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.898118 14952 leader_election.cc:251] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e02fd68b10f84f04a2cbc3945512c242 (127.14.104.60:38345) 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.898869 14919 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02fd68b10f84f04a2cbc3945512c242" is_pre_election: true 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.898924 14857 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" is_pre_election: true 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899006 14919 raft_consensus.cc:2840] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Snoozing failure detection for 1.815s (vote granted) 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899039 14857 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Snoozing failure detection for 2.151s (vote granted) 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899053 14919 raft_consensus.cc:2352] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc86b6e11024fdb8efd341f6b1beb9a in term 0. 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899075 14857 raft_consensus.cc:2352] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc86b6e11024fdb8efd341f6b1beb9a in term 0. 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899457 14768 leader_election.cc:387] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 pre-election: Vote granted by peer 53546bfe72ce43e99dcc08fee1ec6b84 (127.14.104.61:42699) 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899502 14768 leader_election.cc:278] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899708 14766 leader_election.cc:387] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 pre-election: Vote granted by peer e02fd68b10f84f04a2cbc3945512c242 (127.14.104.60:38345) 20:04:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899736 14952 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Snoozing failure detection for 2.003s (election complete) 20:04:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899766 14952 raft_consensus.cc:2646] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 0 FOLLOWER]: Leader pre-election won for term 1 20:04:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899783 14952 raft_consensus.cc:471] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 20:04:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899803 14952 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Snoozing failure detection for 2.052s (starting election) 20:04:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899819 14952 raft_consensus.cc:2892] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 0 FOLLOWER]: Advancing to term 1 20:04:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.899899 14952 consensus_meta.cc:318] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating active role to FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.900985 14952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.901062 14952 leader_election.cc:187] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 election: Running leader election. 20:04:12.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.901195 14952 leader_election.cc:251] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 election: Requesting vote from peer 53546bfe72ce43e99dcc08fee1ec6b84 (127.14.104.61:42699) 20:04:12.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.901293 14952 leader_election.cc:251] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 election: Requesting vote from peer e02fd68b10f84f04a2cbc3945512c242 (127.14.104.60:38345) 20:04:12.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.901799 14857 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" 20:04:12.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.901859 14857 raft_consensus.cc:2892] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 0 FOLLOWER]: Advancing to term 1 20:04:12.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.901934 14857 consensus_meta.cc:318] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Updating active role to FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.901967 14857 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Snoozing failure detection for 2.185s (vote granted) 20:04:12.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.902737 14857 raft_consensus.cc:2352] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc86b6e11024fdb8efd341f6b1beb9a in term 1. 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903069 14768 leader_election.cc:387] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 election: Vote granted by peer 53546bfe72ce43e99dcc08fee1ec6b84 (127.14.104.61:42699) 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903092 14768 leader_election.cc:278] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903247 14952 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Snoozing failure detection for 1.963s (election complete) 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903277 14919 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02fd68b10f84f04a2cbc3945512c242" 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903283 14952 raft_consensus.cc:2646] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 1 FOLLOWER]: Leader election won for term 1 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903331 14919 raft_consensus.cc:2892] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 0 FOLLOWER]: Advancing to term 1 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903354 14952 consensus_meta.cc:318] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating active role to LEADER. Consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903425 14919 consensus_meta.cc:318] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Updating active role to FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903462 14919 raft_consensus.cc:2840] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Snoozing failure detection for 2.072s (vote granted) 20:04:12.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903569 14952 raft_consensus.cc:666] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 1 LEADER]: Becoming Leader. State: Replica: 9bc86b6e11024fdb8efd341f6b1beb9a, State: Running, Role: LEADER 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903877 14952 consensus_queue.cc:206] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.903965 14952 peer_manager.cc:63] Updating peers from new config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.904011 14952 peer_manager.cc:75] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Adding remote peer. Peer: permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.904314 14952 peer_manager.cc:75] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Adding remote peer. Peer: permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.904353 14919 raft_consensus.cc:2352] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc86b6e11024fdb8efd341f6b1beb9a in term 1. 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.904754 14766 leader_election.cc:387] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [CANDIDATE]: Term 1 election: Vote granted by peer e02fd68b10f84f04a2cbc3945512c242 (127.14.104.60:38345) 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.905627 15352 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bc86b6e11024fdb8efd341f6b1beb9a. Latest consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.905658 15352 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: This master's current role is: LEADER 20:04:12.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.905933 15356 catalog_manager.cc:1033] Loading table and tablet metadata into memory... 20:04:12.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.906018 15354 log_cache.cc:237] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating pinned index to 2 20:04:12.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.906260 15354 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 2, LogCacheStats(num_ops=1, bytes=168) 20:04:12.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.907050 15356 catalog_manager.cc:1044] Initializing Kudu internal certificate authority... 20:04:12.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.985692 15354 log_cache.cc:237] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating pinned index to 3 20:04:12.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.985929 15354 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 3, LogCacheStats(num_ops=2, bytes=1963) 20:04:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.986855 14857 raft_consensus.cc:1180] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Refusing update from remote peer 9bc86b6e11024fdb8efd341f6b1beb9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 20:04:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.986860 14919 raft_consensus.cc:1180] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Refusing update from remote peer 9bc86b6e11024fdb8efd341f6b1beb9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 20:04:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.987274 14857 raft_consensus.cc:1014] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: 53546bfe72ce43e99dcc08fee1ec6b84, State: Running, Role: FOLLOWER. Response: responder_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" responder_term: 1 status { last_received { term: 0 index: 0 } last_committed_idx: 0 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } } 20:04:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.987313 14919 raft_consensus.cc:1014] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: e02fd68b10f84f04a2cbc3945512c242, State: Running, Role: FOLLOWER. Response: responder_uuid: "e02fd68b10f84f04a2cbc3945512c242" responder_term: 1 status { last_received { term: 0 index: 0 } last_committed_idx: 0 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } } 20:04:12.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.988018 14952 consensus_queue.cc:976] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 20:04:12.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.988337 15352 consensus_queue.cc:976] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989207 14857 consensus_meta.cc:318] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Updating active role to FOLLOWER. Consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989336 14857 raft_consensus.cc:1448] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Early marking committed up to 0, Last pending opid index: 0, preceding opid index: 0, requested index: 0 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989368 14857 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 0.0 as committed. Now trying to mark 0 which would be a no-op. 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989428 14919 consensus_meta.cc:318] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Updating active role to FOLLOWER. Consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989454 14857 raft_consensus.cc:2205] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Starting consensus round: term: 1 index: 1 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989552 14919 raft_consensus.cc:1448] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Early marking committed up to 0, Last pending opid index: 0, preceding opid index: 0, requested index: 0 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989570 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 0.0 as committed. Now trying to mark 0 which would be a no-op. 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989637 14919 raft_consensus.cc:2205] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Starting consensus round: term: 1 index: 1 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989740 14857 raft_consensus.cc:1039] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Starting transaction: term: 1 index: 2 20:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.989838 14919 raft_consensus.cc:1039] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Starting transaction: term: 1 index: 2 20:04:12.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.991133 14857 raft_consensus.cc:1570] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Marking committed up to 0 20:04:12.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.991133 14919 raft_consensus.cc:1570] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Marking committed up to 0 20:04:12.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.991156 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 0.0 as committed. Now trying to mark 0 which would be a no-op. 20:04:12.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.991156 14857 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 0.0 as committed. Now trying to mark 0 which would be a no-op. 20:04:12.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.991317 14966 sys_catalog.cc:337] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bc86b6e11024fdb8efd341f6b1beb9a. Latest consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.991317 14948 sys_catalog.cc:337] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bc86b6e11024fdb8efd341f6b1beb9a. Latest consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.991336 14948 sys_catalog.cc:340] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.991336 14966 sys_catalog.cc:340] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.993641 15384 log_cache.cc:237] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Updating pinned index to 3 20:04:12.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.993784 15384 log_cache.cc:415] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Evicting log cache: after state: Pinned index: 3, LogCacheStats(num_ops=2, bytes=1963) 20:04:12.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.994488 15385 log_cache.cc:237] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Updating pinned index to 3 20:04:12.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.994613 15385 log_cache.cc:415] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Evicting log cache: after state: Pinned index: 3, LogCacheStats(num_ops=2, bytes=1963) 20:04:12.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.994746 14857 raft_consensus.cc:1014] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: 53546bfe72ce43e99dcc08fee1ec6b84, State: Running, Role: FOLLOWER. Response: responder_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" responder_term: 1 status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } } 20:04:12.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.994824 14919 raft_consensus.cc:1014] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: e02fd68b10f84f04a2cbc3945512c242, State: Running, Role: FOLLOWER. Response: responder_uuid: "e02fd68b10f84f04a2cbc3945512c242" responder_term: 1 status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } } 20:04:12.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.995275 15352 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 0 to 2 20:04:12.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.995348 15352 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 3, LogCacheStats(num_ops=2, bytes=1963) 20:04:12.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.995405 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 2 to 2 20:04:12.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.995440 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 0 to 2 20:04:12.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.995465 14952 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 3, LogCacheStats(num_ops=2, bytes=1963) 20:04:12.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.995942 14952 pending_rounds.cc:171] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Last triggered apply was: 0.0 Starting to apply from log index: 1 20:04:12.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.995961 15390 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.995996 15390 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: This master's current role is: LEADER 20:04:12.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.996085 14952 raft_consensus.cc:2734] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [term 1 LEADER]: Committing NO_OP with op id 1.1 20:04:12.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.996199 14919 raft_consensus.cc:1448] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Early marking committed up to 2, Last pending opid index: 2, preceding opid index: 2, requested index: 2 20:04:12.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.996227 14919 pending_rounds.cc:171] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Last triggered apply was: 0.0 Starting to apply from log index: 1 20:04:12.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.996318 14919 raft_consensus.cc:2734] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Committing NO_OP with op id 1.1 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.996847 14919 raft_consensus.cc:1570] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Marking committed up to 2 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.996896 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 1.2 as committed. Now trying to mark 2 which would be a no-op. 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.996968 14919 raft_consensus.cc:1014] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: e02fd68b10f84f04a2cbc3945512c242, State: Running, Role: FOLLOWER. Response: responder_uuid: "e02fd68b10f84f04a2cbc3945512c242" responder_term: 1 status { last_received { term: 1 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 2 } } 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997018 14952 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997040 14952 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [sys.catalog]: This master's current role is: LEADER 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997133 14948 sys_catalog.cc:337] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997149 14948 sys_catalog.cc:340] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997512 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 2 to 2 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997536 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 2 to 2 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997611 14952 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 3, LogCacheStats(num_ops=1, bytes=1795) 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997624 14857 raft_consensus.cc:1448] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Early marking committed up to 2, Last pending opid index: 2, preceding opid index: 2, requested index: 2 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997650 14857 pending_rounds.cc:171] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Last triggered apply was: 0.0 Starting to apply from log index: 1 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997680 15356 catalog_manager.cc:931] Generated new certificate authority record 20:04:12.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.997735 14857 raft_consensus.cc:2734] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Committing NO_OP with op id 1.1 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998014 15381 mvcc.cc:203] Tried to move safe_time back from 6314833727427051520 to 6314833727096967168. Current Snapshot: MvccSnapshot[committed={T|T < 6314833727427051520}] 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998206 14857 raft_consensus.cc:1570] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Marking committed up to 2 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998236 14857 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 1.2 as committed. Now trying to mark 2 which would be a no-op. 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998294 14857 raft_consensus.cc:1014] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: 53546bfe72ce43e99dcc08fee1ec6b84, State: Running, Role: FOLLOWER. Response: responder_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" responder_term: 1 status { last_received { term: 1 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 2 } } 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998447 14966 sys_catalog.cc:337] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc86b6e11024fdb8efd341f6b1beb9a" member_type: VOTER last_known_addr { host: "127.14.104.62" port: 45145 } } peers { permanent_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" member_type: VOTER last_known_addr { host: "127.14.104.61" port: 42699 } } peers { permanent_uuid: "e02fd68b10f84f04a2cbc3945512c242" member_type: VOTER last_known_addr { host: "127.14.104.60" port: 38345 } } } 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998488 14966 sys_catalog.cc:340] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [sys.catalog]: This master's current role is: FOLLOWER 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998864 15382 mvcc.cc:203] Tried to move safe_time back from 6314833727427051520 to 6314833727096967168. Current Snapshot: MvccSnapshot[committed={T|T < 6314833727427051520}] 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998863 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 2 to 2 20:04:12.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998889 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 2 to 2 20:04:12.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998909 14952 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 3, LogCacheStats(num_ops=1, bytes=1795) 20:04:12.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:12.998975 15356 catalog_manager.cc:1053] Loading token signing keys... 20:04:13.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.006249 15354 log_cache.cc:237] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating pinned index to 4 20:04:13.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.006312 15354 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 2 to 2 20:04:13.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.006331 15354 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 2 to 2 20:04:13.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.006372 15354 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 4, LogCacheStats(num_ops=1, bytes=1043) 20:04:13.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.007436 14919 raft_consensus.cc:1448] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Early marking committed up to 0, Last pending opid index: 0, preceding opid index: 2, requested index: 2 20:04:13.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.007462 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 1.2 as committed. Now trying to mark 0 which would be a no-op. 20:04:13.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.008054 14919 raft_consensus.cc:1039] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Starting transaction: term: 1 index: 3 20:04:13.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.008381 14919 raft_consensus.cc:1570] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Marking committed up to 2 20:04:13.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.008399 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 1.2 as committed. Now trying to mark 2 which would be a no-op. 20:04:13.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.009137 15384 log_cache.cc:237] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Updating pinned index to 4 20:04:13.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.009266 15384 log_cache.cc:415] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Evicting log cache: after state: Pinned index: 4, LogCacheStats(num_ops=1, bytes=1043) 20:04:13.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.009366 14919 raft_consensus.cc:1014] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: e02fd68b10f84f04a2cbc3945512c242, State: Running, Role: FOLLOWER. Response: responder_uuid: "e02fd68b10f84f04a2cbc3945512c242" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 3 } } 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.009821 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 2 to 3 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.009853 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 2 to 2 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.009876 14952 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 4, LogCacheStats(num_ops=1, bytes=1043) 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010051 14952 pending_rounds.cc:171] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Last triggered apply was: 1.2 Starting to apply from log index: 3 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010376 14857 raft_consensus.cc:1448] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Early marking committed up to 0, Last pending opid index: 0, preceding opid index: 2, requested index: 2 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010396 14857 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 1.2 as committed. Now trying to mark 0 which would be a no-op. 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010424 14857 raft_consensus.cc:1039] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Starting transaction: term: 1 index: 3 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010641 14919 raft_consensus.cc:1448] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Early marking committed up to 3, Last pending opid index: 3, preceding opid index: 3, requested index: 3 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010660 14919 pending_rounds.cc:171] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Last triggered apply was: 1.2 Starting to apply from log index: 3 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010746 14857 raft_consensus.cc:1570] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Marking committed up to 2 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010761 14919 raft_consensus.cc:1570] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Marking committed up to 3 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010761 14857 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 1.2 as committed. Now trying to mark 2 which would be a no-op. 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010774 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 1.3 as committed. Now trying to mark 3 which would be a no-op. 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.010834 14919 raft_consensus.cc:1014] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: e02fd68b10f84f04a2cbc3945512c242, State: Running, Role: FOLLOWER. Response: responder_uuid: "e02fd68b10f84f04a2cbc3945512c242" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } } 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011160 15385 log_cache.cc:237] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Updating pinned index to 4 20:04:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011292 15385 log_cache.cc:415] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Evicting log cache: after state: Pinned index: 4, LogCacheStats(num_ops=1, bytes=1043) 20:04:13.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011298 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 3 to 3 20:04:13.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011322 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 2 to 2 20:04:13.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011340 14952 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 4, LogCacheStats(num_ops=1, bytes=1043) 20:04:13.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011377 14857 raft_consensus.cc:1014] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: 53546bfe72ce43e99dcc08fee1ec6b84, State: Running, Role: FOLLOWER. Response: responder_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 3 } } 20:04:13.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011771 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 3 to 3 20:04:13.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011791 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 2 to 3 20:04:13.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.011811 14952 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 4, LogCacheStats(num_ops=1, bytes=1043) 20:04:13.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.012362 14856 raft_consensus.cc:1448] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Early marking committed up to 3, Last pending opid index: 3, preceding opid index: 3, requested index: 3 20:04:13.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.012387 14856 pending_rounds.cc:171] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Last triggered apply was: 1.2 Starting to apply from log index: 3 20:04:13.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.012502 14856 raft_consensus.cc:1570] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Marking committed up to 3 20:04:13.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.012516 14856 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 1.3 as committed. Now trying to mark 3 which would be a no-op. 20:04:13.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.012578 14856 raft_consensus.cc:1014] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: 53546bfe72ce43e99dcc08fee1ec6b84, State: Running, Role: FOLLOWER. Response: responder_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } } 20:04:13.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.012972 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 3 to 3 20:04:13.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.012992 14952 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 3 to 3 20:04:13.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.013010 14952 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 4, LogCacheStats(num_ops=1, bytes=1043) 20:04:13.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.014011 15356 catalog_manager.cc:4082] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Generated new TSK 0 20:04:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015010 15354 log_cache.cc:237] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Updating pinned index to 5 20:04:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015069 15354 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 3 to 3 20:04:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015085 15354 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 3 to 3 20:04:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015120 15354 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 5, LogCacheStats(num_ops=1, bytes=610) 20:04:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015781 14919 raft_consensus.cc:1448] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Early marking committed up to 0, Last pending opid index: 0, preceding opid index: 3, requested index: 3 20:04:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015820 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 1.3 as committed. Now trying to mark 0 which would be a no-op. 20:04:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015856 14919 raft_consensus.cc:1039] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Starting transaction: term: 1 index: 4 20:04:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015875 14857 raft_consensus.cc:1448] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Early marking committed up to 0, Last pending opid index: 0, preceding opid index: 3, requested index: 3 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015894 14857 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 1.3 as committed. Now trying to mark 0 which would be a no-op. 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.015928 14857 raft_consensus.cc:1039] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Starting transaction: term: 1 index: 4 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016149 14919 raft_consensus.cc:1570] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Marking committed up to 3 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016180 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 1.3 as committed. Now trying to mark 3 which would be a no-op. 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016201 14857 raft_consensus.cc:1570] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Marking committed up to 3 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016216 14857 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 1.3 as committed. Now trying to mark 3 which would be a no-op. 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016508 15384 log_cache.cc:237] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Updating pinned index to 5 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016510 15385 log_cache.cc:237] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Updating pinned index to 5 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016583 15384 log_cache.cc:415] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Evicting log cache: after state: Pinned index: 5, LogCacheStats(num_ops=1, bytes=610) 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016587 15385 log_cache.cc:415] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Evicting log cache: after state: Pinned index: 5, LogCacheStats(num_ops=1, bytes=610) 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016669 14857 raft_consensus.cc:1014] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: 53546bfe72ce43e99dcc08fee1ec6b84, State: Running, Role: FOLLOWER. Response: responder_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 4 } } 20:04:13.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.016688 14919 raft_consensus.cc:1014] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: e02fd68b10f84f04a2cbc3945512c242, State: Running, Role: FOLLOWER. Response: responder_uuid: "e02fd68b10f84f04a2cbc3945512c242" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 4 } } 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017084 15390 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 3 to 4 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017118 15390 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 3 to 3 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017140 15390 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 5, LogCacheStats(num_ops=1, bytes=610) 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017222 15352 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 4 to 4 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017254 15352 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 3 to 4 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017274 15352 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 5, LogCacheStats(num_ops=1, bytes=610) 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017432 14952 pending_rounds.cc:171] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Last triggered apply was: 1.3 Starting to apply from log index: 4 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017797 14919 raft_consensus.cc:1448] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Early marking committed up to 4, Last pending opid index: 4, preceding opid index: 4, requested index: 4 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017814 14919 pending_rounds.cc:171] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Last triggered apply was: 1.3 Starting to apply from log index: 4 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017905 14919 raft_consensus.cc:1570] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Marking committed up to 4 20:04:13.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017917 14919 pending_rounds.cc:149] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242: Already marked ops through 1.4 as committed. Now trying to mark 4 which would be a no-op. 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.017977 14919 raft_consensus.cc:1014] T 00000000000000000000000000000000 P e02fd68b10f84f04a2cbc3945512c242 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: e02fd68b10f84f04a2cbc3945512c242, State: Running, Role: FOLLOWER. Response: responder_uuid: "e02fd68b10f84f04a2cbc3945512c242" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } } 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.018354 15390 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 4 to 4 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.018383 15390 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 4 to 4 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.018419 15390 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 5, LogCacheStats(num_ops=0, bytes=0) 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.018618 14857 raft_consensus.cc:1448] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Early marking committed up to 4, Last pending opid index: 4, preceding opid index: 4, requested index: 4 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.018646 14857 pending_rounds.cc:171] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Last triggered apply was: 1.3 Starting to apply from log index: 4 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.018733 14857 raft_consensus.cc:1570] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Marking committed up to 4 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.018755 14857 pending_rounds.cc:149] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84: Already marked ops through 1.4 as committed. Now trying to mark 4 which would be a no-op. 20:04:13.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.018818 14857 raft_consensus.cc:1014] T 00000000000000000000000000000000 P 53546bfe72ce43e99dcc08fee1ec6b84 [term 1 FOLLOWER]: Replica replied to status only request. Replica: Replica: 53546bfe72ce43e99dcc08fee1ec6b84, State: Running, Role: FOLLOWER. Response: responder_uuid: "53546bfe72ce43e99dcc08fee1ec6b84" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } } 20:04:13.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.019309 15390 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated majority_replicated watermark from 4 to 4 20:04:13.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.019341 15390 consensus_queue.cc:857] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a [LEADER]: Updated all_replicated watermark from 4 to 4 20:04:13.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:535) I1108 20:04:13.019359 15390 log_cache.cc:415] T 00000000000000000000000000000000 P 9bc86b6e11024fdb8efd341f6b1beb9a: Evicting log cache: after state: Pinned index: 5, LogCacheStats(num_ops=0, bytes=0) 20:04:13.024 [WARN - New I/O worker #228] (ConnectToCluster.java:278) None of the provided masters 127.14.104.60:38345,127.14.104.61:42699,127.14.104.62:45145 is a leader; will retry java.security.PrivilegedActionException: org.apache.kudu.client.NoLeaderFoundException: Master config (127.14.104.60:38345,127.14.104.61:42699,127.14.104.62:45145) has no leader.20:04:13.043