[==========] Running 14 tests from 2 test cases. [----------] Global test environment set-up. [----------] 12 tests from BlockManagerType/TsRecoveryITest [ RUN ] BlockManagerType/TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks/0 WARNING: Logging before InitGoogleLogging() is written to STDERR I0408 18:33:05.384477 6360 ts_recovery-itest.cc:145] Missing blocks is currently only supported by the log block manager. Exiting early! [ OK ] BlockManagerType/TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks/0 (2 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks/1 I0408 18:33:05.390890 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:05.425447 6366 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:05.425608 6366 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:05.426354 6366 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:05.426378 6366 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:05.426381 6366 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:05.427609 6366 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.6366 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:05.428491 6366 master_main.cc:73] Initializing master server... I0408 18:33:05.428695 6366 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 11511us I0408 18:33:05.428894 6366 fs_manager.cc:260] Metadata directory not provided I0408 18:33:05.428927 6366 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:05.429005 6366 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:05.429030 6366 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:05.430344 6366 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "0615b2dbdbbf454280111cf944b5f390" format_stamp: "Formatted at 2019-04-08 18:33:05 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:05.430542 6366 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "0615b2dbdbbf454280111cf944b5f390" format_stamp: "Formatted at 2019-04-08 18:33:05 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:05.431910 6366 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0408 18:33:05.432209 6366 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:05.432245 6366 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:05.432881 6366 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:05.432920 6366 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/wal uuid: "0615b2dbdbbf454280111cf944b5f390" format_stamp: "Formatted at 2019-04-08 18:33:05 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:05.432950 6366 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:05.459302 6366 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:05.459379 6366 master_main.cc:76] Starting Master server... I0408 18:33:05.459522 6381 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:05.459563 6381 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:05.459571 6381 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:05.462659 6366 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:42051 I0408 18:33:05.462751 6366 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:05.462767 6366 webserver.cc:184] Document root disabled I0408 18:33:05.462966 6366 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:41049/ I0408 18:33:05.463207 6366 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:05.463647 6425 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:05.463862 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 6366 I0408 18:33:05.464120 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:42051 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0408 18:33:05.465762 6425 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390: Bootstrap starting. I0408 18:33:05.466177 6425 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390: No blocks or log segments found. Creating new log. I0408 18:33:05.466398 6425 log.cc:520] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390: Log is configured to *not* fsync() on all Append() calls I0408 18:33:05.467406 6425 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390: No bootstrap required, opened a new log I0408 18:33:05.468679 6425 raft_consensus.cc:304] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0615b2dbdbbf454280111cf944b5f390" member_type: VOTER } I0408 18:33:05.468721 6425 raft_consensus.cc:330] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:05.468744 6425 raft_consensus.cc:605] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0615b2dbdbbf454280111cf944b5f390, State: Initialized, Role: FOLLOWER I0408 18:33:05.468865 6425 consensus_queue.cc:226] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0615b2dbdbbf454280111cf944b5f390" member_type: VOTER } I0408 18:33:05.468892 6425 raft_consensus.cc:344] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:05.468907 6425 raft_consensus.cc:436] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:05.468940 6425 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390: Snoozing failure detection for 1.512s (starting election) I0408 18:33:05.468961 6425 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:05.469364 6425 raft_consensus.cc:458] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0615b2dbdbbf454280111cf944b5f390" member_type: VOTER } I0408 18:33:05.469427 6425 leader_election.cc:258] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:05.469588 6428 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390: Snoozing failure detection for 1.509s (election complete) I0408 18:33:05.469621 6428 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:05.469702 6428 raft_consensus.cc:572] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [term 1 LEADER]: Becoming Leader. State: Replica: 0615b2dbdbbf454280111cf944b5f390, State: Running, Role: LEADER I0408 18:33:05.469780 6428 consensus_queue.cc:203] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0615b2dbdbbf454280111cf944b5f390" member_type: VOTER } I0408 18:33:05.470000 6428 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0615b2dbdbbf454280111cf944b5f390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0615b2dbdbbf454280111cf944b5f390" member_type: VOTER } } I0408 18:33:05.470031 6428 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [sys.catalog]: This master's current role is: LEADER I0408 18:33:05.470131 6428 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0615b2dbdbbf454280111cf944b5f390. Latest consensus state: current_term: 1 leader_uuid: "0615b2dbdbbf454280111cf944b5f390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0615b2dbdbbf454280111cf944b5f390" member_type: VOTER } } I0408 18:33:05.470151 6428 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [sys.catalog]: This master's current role is: LEADER I0408 18:33:05.470213 6425 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "0615b2dbdbbf454280111cf944b5f390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0615b2dbdbbf454280111cf944b5f390" member_type: VOTER } } I0408 18:33:05.470235 6425 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [sys.catalog]: This master's current role is: LEADER I0408 18:33:05.470332 6425 sys_catalog.cc:422] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:05.470566 6366 master_main.cc:79] Master server successfully started. W0408 18:33:05.470794 6433 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:05.470829 6433 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:05.475224 6432 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:05.475364 6432 catalog_manager.cc:983] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:05.505625 6426 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:05.505789 6426 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:05.506449 6426 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:05.506472 6426 flags.cc:412] Enabled experimental flag: --flush_threshold_mb=1 W0408 18:33:05.506476 6426 flags.cc:412] Enabled experimental flag: --flush_threshold_secs=1 W0408 18:33:05.506484 6426 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:05.507526 6426 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.6.54.62:42051 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.6426 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:05.508462 6426 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:05.508679 6426 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 11511us I0408 18:33:05.508812 6426 fs_manager.cc:260] Metadata directory not provided I0408 18:33:05.508842 6426 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:05.508898 6426 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:05.508914 6426 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:05.510174 6426 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "e0de2f3696ba40e092c7bb36723895ce" format_stamp: "Formatted at 2019-04-08 18:33:05 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:05.510334 6426 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "e0de2f3696ba40e092c7bb36723895ce" format_stamp: "Formatted at 2019-04-08 18:33:05 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:05.511653 6426 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s I0408 18:33:05.511983 6426 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:05.512022 6426 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:05.512836 6426 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:05.512878 6426 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "e0de2f3696ba40e092c7bb36723895ce" format_stamp: "Formatted at 2019-04-08 18:33:05 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:05.512918 6426 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:05.560245 6426 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:05.560432 6426 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:05.560495 6426 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:05.565420 6426 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:33437 I0408 18:33:05.565503 6426 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:05.565521 6426 webserver.cc:184] Document root disabled I0408 18:33:05.565793 6426 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:35753/ I0408 18:33:05.566063 6426 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:05.566490 6426 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:05.566910 6554 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:05.566936 6554 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:05.566941 6554 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:05.568969 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 6426 I0408 18:33:05.573413 6432 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:05.574223 6432 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:05.574895 6553 heartbeater.cc:346] Connected to a master server at 127.6.54.62:42051 I0408 18:33:05.574940 6553 heartbeater.cc:426] Registering TS with master... I0408 18:33:05.575042 6553 heartbeater.cc:475] Master 127.6.54.62:42051 requested a full tablet report, sending... I0408 18:33:05.575515 6392 ts_manager.cc:81] Registered new tserver with Master: e0de2f3696ba40e092c7bb36723895ce (127.6.54.1:33437) I0408 18:33:05.576639 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:05.592120 6432 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 0615b2dbdbbf454280111cf944b5f390: Generated new TSK 0 I0408 18:33:05.594116 6360 test_util.cc:200] Using random seed: 289213186 I0408 18:33:05.601269 6392 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38294: name: "foo" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:05.606468 6475 tablet_service.cc:758] Processing CreateTablet for tablet 4754065036374334a11711d7289d68ec (table=foo [id=fdd4ccd341e54f61b24356f5e54cda03]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:05.606669 6475 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 4754065036374334a11711d7289d68ec. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:05.608325 6475 ts_tablet_manager.cc:1080] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:05.611217 6573 ts_tablet_manager.cc:938] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Bootstrapping tablet I0408 18:33:05.611297 6573 tablet_bootstrap.cc:436] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Bootstrap starting. I0408 18:33:05.611702 6573 tablet_bootstrap.cc:581] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: No blocks or log segments found. Creating new log. I0408 18:33:05.611987 6573 log.cc:520] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Log is configured to *not* fsync() on all Append() calls I0408 18:33:05.612397 6573 tablet_bootstrap.cc:436] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: No bootstrap required, opened a new log I0408 18:33:05.612449 6573 ts_tablet_manager.cc:955] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:05.613435 6573 raft_consensus.cc:304] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2f3696ba40e092c7bb36723895ce" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 33437 } } I0408 18:33:05.613474 6573 raft_consensus.cc:330] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:05.613487 6573 raft_consensus.cc:605] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de2f3696ba40e092c7bb36723895ce, State: Initialized, Role: FOLLOWER I0408 18:33:05.613634 6573 consensus_queue.cc:226] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2f3696ba40e092c7bb36723895ce" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 33437 } } I0408 18:33:05.613657 6573 raft_consensus.cc:344] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:05.613667 6573 raft_consensus.cc:436] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:05.613684 6573 raft_consensus.cc:2700] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Snoozing failure detection for 1.597s (starting election) I0408 18:33:05.613693 6573 raft_consensus.cc:2752] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:05.614125 6573 raft_consensus.cc:458] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2f3696ba40e092c7bb36723895ce" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 33437 } } I0408 18:33:05.614176 6573 leader_election.cc:258] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:05.614372 6573 ts_tablet_manager.cc:983] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:05.615219 6575 raft_consensus.cc:2700] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Snoozing failure detection for 1.652s (election complete) I0408 18:33:05.615247 6575 raft_consensus.cc:2507] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:05.615320 6575 raft_consensus.cc:572] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [term 1 LEADER]: Becoming Leader. State: Replica: e0de2f3696ba40e092c7bb36723895ce, State: Running, Role: LEADER I0408 18:33:05.615393 6575 consensus_queue.cc:203] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2f3696ba40e092c7bb36723895ce" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 33437 } } I0408 18:33:05.615636 6392 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:60759 I0408 18:33:05.616338 6553 heartbeater.cc:467] Master 127.6.54.62:42051 was elected leader, sending a full tablet report... I0408 18:33:05.616765 6392 catalog_manager.cc:3595] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce reported cstate change: term changed from 0 to 1, leader changed from to e0de2f3696ba40e092c7bb36723895ce (127.6.54.1). New cstate: current_term: 1 leader_uuid: "e0de2f3696ba40e092c7bb36723895ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2f3696ba40e092c7bb36723895ce" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 33437 } health_report { overall_health: HEALTHY } } } I0408 18:33:05.817246 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:05.818349 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:05.818379 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:05.818399 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:05.818408 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:05.818439 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249388306157568 or (T in {6368249388306157568})}] I0408 18:33:05.819977 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:05.845969 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:05.846035 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:05.846045 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249388422430720 or (T in {6368249388422430720})}] I0408 18:33:05.850373 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 23050 rows (656558 bytes) I0408 18:33:05.850587 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.033s user 0.026s sys 0.004s Metrics: {"cfile_init":1,"compiler_manager_pool.queue_time_us":428,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":137,"data dirs.run_wall_time_us":687,"lbm_read_time_us":28,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1027,"lbm_writes_lt_1ms":47,"thread_start_us":84,"threads_started":1} I0408 18:33:05.850716 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 223840 bytes on disk I0408 18:33:05.850847 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} I0408 18:33:06.101027 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:06.101241 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:06.101263 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:06.101269 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:06.101274 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:06.101284 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249389468889088 or (T in {6368249389468889088})}] I0408 18:33:06.101414 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:06.134440 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:06.134495 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:06.134507 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249389601951744 or (T in {6368249389601951744})}] I0408 18:33:06.140556 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 29850 rows (848149 bytes) I0408 18:33:06.140696 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.040s user 0.028s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":21,"data dirs.run_cpu_time_us":92,"data dirs.run_wall_time_us":742,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1398,"lbm_writes_lt_1ms":55} I0408 18:33:06.140813 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989964 I0408 18:33:06.140887 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:06.140902 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:06.140910 6439 compaction.cc:917] RowSet(0)(current size on disk: ~880398 bytes) I0408 18:33:06.140915 6439 compaction.cc:917] RowSet(1)(current size on disk: ~1140249 bytes) I0408 18:33:06.140924 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249389628923904 or (T in {6368249389628923904})}] I0408 18:33:06.141062 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:06.182718 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:06.182777 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:06.182790 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249389802086400 or (T in {6368249389802086400})}] I0408 18:33:06.194340 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 52900 rows (1493495 bytes) I0408 18:33:06.194412 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.054s user 0.052s sys 0.000s Metrics: {"cfile_cache_hit":12,"cfile_cache_hit_bytes":662215,"cfile_cache_miss":76,"cfile_cache_miss_bytes":3113887,"cfile_init":6,"data dirs.queue_time_us":33,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":1053,"delta_iterators_relevant":4,"lbm_read_time_us":1430,"lbm_reads_lt_1ms":100,"lbm_write_time_us":3157,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":79} I0408 18:33:06.194494 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 539558 bytes on disk I0408 18:33:06.194628 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4} I0408 18:33:06.444871 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=2.929465 I0408 18:33:06.445058 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:06.445071 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:06.445077 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:06.445083 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 4120343 bytes I0408 18:33:06.445092 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249390877634560 or (T in {6368249390877634560})}] I0408 18:33:06.445246 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:06.481189 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:06.481315 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:06.481379 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249391023874048 or (T in {6368249391023874048})}] I0408 18:33:06.488121 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 40800 rows (1155224 bytes) I0408 18:33:06.488327 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.043s user 0.039s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":22,"data dirs.run_cpu_time_us":75,"data dirs.run_wall_time_us":1630,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1848,"lbm_writes_lt_1ms":66,"spinlock_wait_cycles":896} I0408 18:33:06.488440 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989982 I0408 18:33:06.488509 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:06.488523 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:06.488529 6439 compaction.cc:917] RowSet(2)(current size on disk: ~2033053 bytes) I0408 18:33:06.488534 6439 compaction.cc:917] RowSet(3)(current size on disk: ~1559779 bytes) I0408 18:33:06.488555 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249391055290368}] I0408 18:33:06.488672 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:06.558812 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:06.560047 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:06.561306 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249391342161920 or (T in {6368249391342161920})}] I0408 18:33:06.584007 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 93700 rows (2635273 bytes) I0408 18:33:06.584704 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.096s user 0.079s sys 0.008s Metrics: {"cfile_cache_hit":13,"cfile_cache_hit_bytes":940810,"cfile_cache_miss":117,"cfile_cache_miss_bytes":5790200,"cfile_init":6,"data dirs.queue_time_us":87,"data dirs.run_cpu_time_us":533,"data dirs.run_wall_time_us":5606,"delta_iterators_relevant":4,"lbm_read_time_us":2877,"lbm_reads_lt_1ms":141,"lbm_write_time_us":6557,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":128} I0408 18:33:06.584794 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 972533 bytes on disk I0408 18:33:06.584908 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4} I0408 18:33:06.584962 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=0.945090 I0408 18:33:06.585037 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:06.585050 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:06.585057 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:06.585062 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 2039575 bytes I0408 18:33:06.585069 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249391451750400}] I0408 18:33:06.585165 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:06.598557 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:06.598855 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:06.598876 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249391506395136 or (T in {6368249391506735104,6368249391506395136})}] I0408 18:33:06.601533 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 12850 rows (369407 bytes) I0408 18:33:06.601670 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":21,"data dirs.run_cpu_time_us":62,"data dirs.run_wall_time_us":608,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":35} I0408 18:33:06.601753 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 120592 bytes on disk I0408 18:33:06.601912 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4} I0408 18:33:06.601960 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989994 I0408 18:33:06.602002 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:06.602020 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:06.602030 6439 compaction.cc:917] RowSet(4)(current size on disk: ~3607806 bytes) I0408 18:33:06.602056 6439 compaction.cc:917] RowSet(5)(current size on disk: ~489999 bytes) I0408 18:33:06.602067 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249391520256000}] I0408 18:33:06.602151 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:06.651221 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:06.651324 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:06.651360 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249391720144896 or (T in {6368249391720144896})}] I0408 18:33:06.664609 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 106550 rows (2995328 bytes) I0408 18:33:06.664721 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.063s user 0.060s sys 0.000s Metrics: {"cfile_cache_hit":12,"cfile_cache_hit_bytes":659976,"cfile_cache_miss":138,"cfile_cache_miss_bytes":7015060,"cfile_init":5,"data dirs.queue_time_us":68,"data dirs.run_cpu_time_us":170,"data dirs.run_wall_time_us":5356,"delta_iterators_relevant":4,"lbm_read_time_us":2315,"lbm_reads_lt_1ms":158,"lbm_write_time_us":3729,"lbm_writes_lt_1ms":144,"spinlock_wait_cycles":768} I0408 18:33:06.664788 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 1107891 bytes on disk I0408 18:33:06.664873 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0408 18:33:06.915021 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=2.929465 I0408 18:33:06.915102 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:06.915117 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:06.915120 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:06.915123 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 4120343 bytes I0408 18:33:06.915132 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249392802504704 or (T in {6368249392802504704})}] I0408 18:33:06.915259 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:06.939842 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:06.939941 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:06.939990 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249392903692288 or (T in {6368249392903692288})}] I0408 18:33:06.945327 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 39600 rows (1124485 bytes) I0408 18:33:06.945894 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.031s user 0.025s sys 0.003s Metrics: {"cfile_init":1,"data dirs.queue_time_us":46,"data dirs.run_cpu_time_us":245,"data dirs.run_wall_time_us":2900,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1800,"lbm_writes_lt_1ms":66} I0408 18:33:06.946018 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 396216 bytes on disk I0408 18:33:06.946154 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} I0408 18:33:06.946272 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989993 I0408 18:33:06.946497 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:06.946529 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:06.946553 6439 compaction.cc:917] RowSet(6)(current size on disk: ~4103219 bytes) I0408 18:33:06.946684 6439 compaction.cc:917] RowSet(7)(current size on disk: ~1520701 bytes) I0408 18:33:06.946712 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249392931696640 or (T in {6368249392931696640})}] I0408 18:33:06.946799 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.016144 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.016264 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.016304 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249393217523712}] I0408 18:33:07.036718 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 146150 rows (4097885 bytes) I0408 18:33:07.037468 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.091s user 0.073s sys 0.016s Metrics: {"cfile_cache_hit":15,"cfile_cache_hit_bytes":1214323,"cfile_cache_miss":179,"cfile_cache_miss_bytes":9315227,"cfile_init":5,"data dirs.queue_time_us":59,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":6789,"delta_iterators_relevant":4,"lbm_read_time_us":3206,"lbm_reads_lt_1ms":199,"lbm_write_time_us":6396,"lbm_writes_lt_1ms":192} I0408 18:33:07.038385 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=0.945090 I0408 18:33:07.038904 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:07.038933 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:07.038939 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:07.038944 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 2039575 bytes I0408 18:33:07.038950 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249393308684288}] I0408 18:33:07.039094 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.050240 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.050468 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.050494 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249393355509760 or (T in {6368249393355509760})}] I0408 18:33:07.052995 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 12250 rows (353466 bytes) I0408 18:33:07.053133 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":164,"data dirs.run_cpu_time_us":84,"data dirs.run_wall_time_us":524,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":35} I0408 18:33:07.053278 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 1650714 bytes on disk I0408 18:33:07.053653 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":18,"lbm_reads_lt_1ms":8} I0408 18:33:07.053755 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989900 I0408 18:33:07.053823 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:07.053836 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:07.053843 6439 compaction.cc:917] RowSet(8)(current size on disk: ~5633861 bytes) I0408 18:33:07.053848 6439 compaction.cc:917] RowSet(9)(current size on disk: ~468204 bytes) I0408 18:33:07.053856 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249393372045312}] I0408 18:33:07.054219 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.171648 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.171782 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.171835 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249393850621952 or (T in {6368249393850621952})}] I0408 18:33:07.201118 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 158400 rows (4438304 bytes) I0408 18:33:07.201252 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.147s user 0.127s sys 0.015s Metrics: {"cfile_cache_hit":12,"cfile_cache_hit_bytes":841337,"cfile_cache_miss":202,"cfile_cache_miss_bytes":10586807,"cfile_init":5,"data dirs.queue_time_us":20,"data dirs.run_cpu_time_us":72,"data dirs.run_wall_time_us":8834,"delta_iterators_relevant":4,"lbm_read_time_us":4339,"lbm_reads_lt_1ms":222,"lbm_write_time_us":7895,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":205,"spinlock_wait_cycles":3328} I0408 18:33:07.201442 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=0.945090 I0408 18:33:07.201565 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:07.201628 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:07.201673 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:07.201699 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 2039575 bytes I0408 18:33:07.201709 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249393974210560 or (T in {6368249393974210560})}] I0408 18:33:07.201992 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.218446 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.218571 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.218598 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249394044919808 or (T in {6368249394044919808})}] I0408 18:33:07.221715 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 18200 rows (517489 bytes) I0408 18:33:07.221851 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.020s user 0.019s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":29,"data dirs.run_cpu_time_us":121,"data dirs.run_wall_time_us":613,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1159,"lbm_writes_lt_1ms":42} I0408 18:33:07.221984 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989894 I0408 18:33:07.222328 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:07.222388 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:07.222434 6439 compaction.cc:917] RowSet(10)(current size on disk: ~6113808 bytes) I0408 18:33:07.222472 6439 compaction.cc:917] RowSet(11)(current size on disk: ~691386 bytes) I0408 18:33:07.222504 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249394056933376}] I0408 18:33:07.222659 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.351886 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.351948 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.351958 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249394588901376 or (T in {6368249394588901376})}] I0408 18:33:07.384188 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 176600 rows (4947657 bytes) I0408 18:33:07.384409 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.162s user 0.149s sys 0.012s Metrics: {"cfile_cache_hit":16,"cfile_cache_hit_bytes":1241583,"cfile_cache_miss":220,"cfile_cache_miss_bytes":11510425,"cfile_init":7,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":99,"data dirs.run_wall_time_us":11546,"delta_iterators_relevant":4,"lbm_read_time_us":4678,"lbm_reads_lt_1ms":248,"lbm_write_time_us":9544,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":226} I0408 18:33:07.384577 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=0.945090 I0408 18:33:07.384671 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:07.384711 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:07.384735 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:07.384759 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 2039575 bytes I0408 18:33:07.384812 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249394723684352 or (T in {6368249394723684352})}] I0408 18:33:07.384932 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.399608 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.399767 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.399823 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249394786623488 or (T in {6368249394786623488})}] I0408 18:33:07.402506 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 19600 rows (558380 bytes) I0408 18:33:07.402683 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.018s user 0.017s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":102,"data dirs.run_wall_time_us":653,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":44,"spinlock_wait_cycles":1792} I0408 18:33:07.402792 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989881 I0408 18:33:07.402871 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:07.402887 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:07.402891 6439 compaction.cc:917] RowSet(12)(current size on disk: ~6829422 bytes) I0408 18:33:07.402895 6439 compaction.cc:917] RowSet(13)(current size on disk: ~746108 bytes) I0408 18:33:07.402900 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249394800984064}] I0408 18:33:07.402992 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.489656 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.489989 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.490023 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249395156865024 or (T in {6368249395156865024})}] I0408 18:33:07.514533 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 196200 rows (5488636 bytes) I0408 18:33:07.514834 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.112s user 0.095s sys 0.016s Metrics: {"cfile_cache_hit":13,"cfile_cache_hit_bytes":1039997,"cfile_cache_miss":245,"cfile_cache_miss_bytes":13153837,"cfile_init":7,"data dirs.queue_time_us":33,"data dirs.run_cpu_time_us":137,"data dirs.run_wall_time_us":14268,"delta_iterators_relevant":4,"lbm_read_time_us":4804,"lbm_reads_lt_1ms":273,"lbm_write_time_us":7112,"lbm_writes_lt_1ms":250,"spinlock_wait_cycles":1792} I0408 18:33:07.515030 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=0.945090 I0408 18:33:07.516657 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:07.517304 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:07.517328 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:07.517335 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 2039575 bytes I0408 18:33:07.517344 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249395265314816 or (T in {6368249395265314816})}] I0408 18:33:07.517443 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.533006 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.533118 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.533180 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249395334242304 or (T in {6368249395334242304})}] I0408 18:33:07.535773 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 14600 rows (418612 bytes) I0408 18:33:07.535872 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.021s user 0.017s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":19,"data dirs.run_cpu_time_us":74,"data dirs.run_wall_time_us":568,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":816,"lbm_writes_lt_1ms":38} I0408 18:33:07.536130 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 2239057 bytes on disk I0408 18:33:07.536311 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":25,"lbm_reads_lt_1ms":8} I0408 18:33:07.536424 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989888 I0408 18:33:07.536492 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:07.536510 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:07.536517 6439 compaction.cc:917] RowSet(14)(current size on disk: ~7589516 bytes) I0408 18:33:07.536522 6439 compaction.cc:917] RowSet(15)(current size on disk: ~556789 bytes) I0408 18:33:07.536530 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249395346321408 or (T in {6368249395346321408})}] I0408 18:33:07.536643 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.684906 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.684957 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.684967 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249395956158464 or (T in {6368249395956158464})}] I0408 18:33:07.722509 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 210800 rows (5895987 bytes) I0408 18:33:07.722643 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.186s user 0.165s sys 0.020s Metrics: {"cfile_cache_hit":14,"cfile_cache_hit_bytes":1077468,"cfile_cache_miss":262,"cfile_cache_miss_bytes":14192024,"cfile_init":5,"data dirs.queue_time_us":22,"data dirs.run_cpu_time_us":91,"data dirs.run_wall_time_us":18919,"delta_iterators_relevant":4,"lbm_read_time_us":5818,"lbm_reads_lt_1ms":282,"lbm_write_time_us":11175,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":267} I0408 18:33:07.722754 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 2263657 bytes on disk I0408 18:33:07.722872 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0408 18:33:07.722940 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:07.723059 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:07.723090 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:07.723124 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:07.723143 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:07.723204 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249396110712832 or (T in {6368249396110712832})}] I0408 18:33:07.723320 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.742311 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.742365 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.742377 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249396191383552}] I0408 18:33:07.745276 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 21250 rows (605593 bytes) I0408 18:33:07.745440 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.022s user 0.018s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":78,"data dirs.run_wall_time_us":631,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1301,"lbm_writes_lt_1ms":45} I0408 18:33:07.745594 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 204303 bytes on disk I0408 18:33:07.745739 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4} I0408 18:33:07.746098 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989904 I0408 18:33:07.746173 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:07.746194 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:07.746201 6439 compaction.cc:917] RowSet(16)(current size on disk: ~8159644 bytes) I0408 18:33:07.746206 6439 compaction.cc:917] RowSet(17)(current size on disk: ~809896 bytes) I0408 18:33:07.746213 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249396206952448 or (T in {6368249396206952448})}] I0408 18:33:07.746383 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.900725 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:07.908141 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000002 I0408 18:33:07.911999 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.912047 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.912057 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249396885123072 or (T in {6368249396885123072})}] I0408 18:33:07.959237 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 232050 rows (6485471 bytes) I0408 18:33:07.959527 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.213s user 0.194s sys 0.017s Metrics: {"cfile_cache_hit":15,"cfile_cache_hit_bytes":1265130,"cfile_cache_miss":287,"cfile_cache_miss_bytes":15543680,"cfile_init":5,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":195,"data dirs.run_wall_time_us":23847,"delta_iterators_relevant":4,"lbm_read_time_us":6779,"lbm_reads_lt_1ms":307,"lbm_write_time_us":12307,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":291,"spinlock_wait_cycles":575360} I0408 18:33:07.959758 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:07.960086 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:07.960142 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:07.960186 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:07.960227 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:07.960274 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249397081911296}] I0408 18:33:07.960904 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:07.983743 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:07.984118 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:07.984216 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249397179711488 or (T in {6368249397179711488})}] I0408 18:33:07.987697 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 23400 rows (665486 bytes) I0408 18:33:07.987865 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.028s user 0.024s sys 0.003s Metrics: {"cfile_init":1,"data dirs.queue_time_us":18,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":699,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1518,"lbm_writes_lt_1ms":47} I0408 18:33:07.987963 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8408600 bytes of WAL I0408 18:33:07.988023 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:07.988126 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000001 (ops 1-5003) I0408 18:33:07.990093 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {} I0408 18:33:07.990376 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 2726622 bytes on disk I0408 18:33:07.990558 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":28,"lbm_reads_lt_1ms":8} I0408 18:33:07.990679 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989935 I0408 18:33:07.990818 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:07.990854 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:07.990875 6439 compaction.cc:917] RowSet(18)(current size on disk: ~8983817 bytes) I0408 18:33:07.990888 6439 compaction.cc:917] RowSet(19)(current size on disk: ~893762 bytes) I0408 18:33:07.990896 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249397207441408}] I0408 18:33:07.991118 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:08.173790 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:08.173841 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:08.173851 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249397956501504 or (T in {6368249397956501504})}] I0408 18:33:08.222420 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 255450 rows (7135205 bytes) I0408 18:33:08.222554 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.232s user 0.218s sys 0.012s Metrics: {"cfile_cache_hit":16,"cfile_cache_hit_bytes":1280196,"cfile_cache_miss":310,"cfile_cache_miss_bytes":17237590,"cfile_init":5,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":85,"data dirs.run_wall_time_us":27936,"delta_iterators_relevant":4,"lbm_read_time_us":7822,"lbm_reads_lt_1ms":330,"lbm_write_time_us":11442,"lbm_writes_lt_1ms":320,"spinlock_wait_cycles":1152} I0408 18:33:08.222759 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 2760280 bytes on disk I0408 18:33:08.222913 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} I0408 18:33:08.223018 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:08.223340 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:08.223390 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:08.223421 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:08.223453 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:08.223495 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249398159155200 or (T in {6368249398159155200})}] I0408 18:33:08.224161 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:08.246233 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:08.246531 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:08.246632 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249398255026176 or (T in {6368249398255439872,6368249398255026176})}] I0408 18:33:08.251194 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 26150 rows (745596 bytes) I0408 18:33:08.251307 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.028s user 0.027s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":22,"data dirs.run_cpu_time_us":91,"data dirs.run_wall_time_us":695,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1897,"lbm_writes_lt_1ms":51} I0408 18:33:08.251461 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989803 I0408 18:33:08.251523 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:08.251595 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:08.251646 6439 compaction.cc:917] RowSet(20)(current size on disk: ~9895485 bytes) I0408 18:33:08.251690 6439 compaction.cc:917] RowSet(21)(current size on disk: ~999376 bytes) I0408 18:33:08.251726 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249398275829760 or (T in {6368249398275829760})}] I0408 18:33:08.251895 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:08.446413 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:08.446465 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:08.446476 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249399073230848 or (T in {6368249399073230848})}] I0408 18:33:08.497649 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 281600 rows (7858233 bytes) I0408 18:33:08.497853 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.246s user 0.217s sys 0.028s Metrics: {"cfile_cache_hit":18,"cfile_cache_hit_bytes":1735529,"cfile_cache_miss":340,"cfile_cache_miss_bytes":18685023,"cfile_init":6,"data dirs.queue_time_us":73,"data dirs.run_cpu_time_us":85,"data dirs.run_wall_time_us":33524,"delta_iterators_relevant":4,"lbm_read_time_us":7976,"lbm_reads_lt_1ms":364,"lbm_write_time_us":14318,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":351,"spinlock_wait_cycles":621568} I0408 18:33:08.498005 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 3058287 bytes on disk I0408 18:33:08.498148 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} I0408 18:33:08.498205 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:08.498296 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:08.498338 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:08.498375 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:08.498407 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:08.498450 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249399288348672}] I0408 18:33:08.498617 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:08.524888 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:08.524945 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:08.524955 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249399393255424 or (T in {6368249399393255424})}] I0408 18:33:08.528882 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 27650 rows (788823 bytes) I0408 18:33:08.528985 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.031s user 0.028s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":55,"data dirs.run_cpu_time_us":79,"data dirs.run_wall_time_us":574,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1618,"lbm_writes_lt_1ms":52} I0408 18:33:08.529084 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 269213 bytes on disk I0408 18:33:08.529515 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4} I0408 18:33:08.529661 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989727 I0408 18:33:08.529741 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:08.529762 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:08.529769 6439 compaction.cc:917] RowSet(22)(current size on disk: ~10916520 bytes) I0408 18:33:08.529773 6439 compaction.cc:917] RowSet(23)(current size on disk: ~1058036 bytes) I0408 18:33:08.529783 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249399416197120 or (T in {6368249399416197120})}] I0408 18:33:08.529880 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:08.744310 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:08.744452 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:08.744511 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249400294023168 or (T in {6368249400294023168})}] I0408 18:33:08.794872 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 309250 rows (8625231 bytes) I0408 18:33:08.794952 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.265s user 0.247s sys 0.016s Metrics: {"cfile_cache_hit":14,"cfile_cache_hit_bytes":1105368,"cfile_cache_miss":378,"cfile_cache_miss_bytes":21342754,"cfile_init":5,"data dirs.queue_time_us":131,"data dirs.run_cpu_time_us":144,"data dirs.run_wall_time_us":45969,"delta_iterators_relevant":4,"lbm_read_time_us":8643,"lbm_reads_lt_1ms":398,"lbm_write_time_us":15364,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":383,"spinlock_wait_cycles":6144} I0408 18:33:08.795120 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:08.795248 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:08.795271 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:08.795277 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:08.795282 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:08.795290 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249400503017472 or (T in {6368249400503017472})}] I0408 18:33:08.795387 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:08.821141 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:08.822026 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:08.822293 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249400609308672 or (T in {6368249400611131392,6368249400609308672})}] I0408 18:33:08.828696 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 28750 rows (815930 bytes) I0408 18:33:08.828857 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.034s user 0.028s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":19,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":1338,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2160,"lbm_writes_lt_1ms":54} I0408 18:33:08.829032 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 3651790 bytes on disk I0408 18:33:08.829176 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":22,"lbm_reads_lt_1ms":8} I0408 18:33:08.829241 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989835 I0408 18:33:08.829314 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:08.829358 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:08.829391 6439 compaction.cc:917] RowSet(24)(current size on disk: ~11996141 bytes) I0408 18:33:08.829418 6439 compaction.cc:917] RowSet(25)(current size on disk: ~1096810 bytes) I0408 18:33:08.829447 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249400641265664 or (T in {6368249400641265664})}] I0408 18:33:08.829583 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:09.067876 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:09.068194 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:09.068254 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249401620242432 or (T in {6368249401620242432})}] I0408 18:33:09.128952 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 338000 rows (9422913 bytes) I0408 18:33:09.129097 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.300s user 0.279s sys 0.012s Metrics: {"cfile_cache_hit":16,"cfile_cache_hit_bytes":1387941,"cfile_cache_miss":412,"cfile_cache_miss_bytes":23165149,"cfile_init":5,"data dirs.queue_time_us":173,"data dirs.run_cpu_time_us":101,"data dirs.run_wall_time_us":46632,"delta_iterators_relevant":4,"lbm_read_time_us":8984,"lbm_reads_lt_1ms":432,"lbm_write_time_us":15544,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":418} I0408 18:33:09.129190 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=2.929465 I0408 18:33:09.129467 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:09.129487 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:09.129493 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:09.129498 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 4120343 bytes I0408 18:33:09.129506 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249401870979072 or (T in {6368249401870979072})}] I0408 18:33:09.129619 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:09.156311 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:09.156715 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:09.157460 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249401984057344 or (T in {6368249401984057344})}] I0408 18:33:09.163784 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 32100 rows (908705 bytes) I0408 18:33:09.163980 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.035s user 0.023s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":22,"data dirs.run_cpu_time_us":84,"data dirs.run_wall_time_us":1250,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1954,"lbm_writes_lt_1ms":57} I0408 18:33:09.164075 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 4011051 bytes on disk I0408 18:33:09.164249 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":31,"lbm_reads_lt_1ms":8} I0408 18:33:09.164336 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989931 I0408 18:33:09.164438 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:09.164752 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:09.164774 6439 compaction.cc:917] RowSet(26)(current size on disk: ~13115350 bytes) I0408 18:33:09.164780 6439 compaction.cc:917] RowSet(27)(current size on disk: ~1227319 bytes) I0408 18:33:09.164789 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249402017054720 or (T in {6368249402017054720})}] I0408 18:33:09.164896 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:09.421288 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:09.421377 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:09.421391 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249403068555264 or (T in {6368249403068555264})}] I0408 18:33:09.486994 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 370100 rows (10313603 bytes) I0408 18:33:09.487124 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.323s user 0.281s sys 0.040s Metrics: {"cfile_cache_hit":17,"cfile_cache_hit_bytes":1601867,"cfile_cache_miss":449,"cfile_cache_miss_bytes":25294113,"cfile_init":5,"data dirs.queue_time_us":46,"data dirs.run_cpu_time_us":91,"data dirs.run_wall_time_us":52563,"delta_iterators_relevant":4,"lbm_read_time_us":9822,"lbm_reads_lt_1ms":469,"lbm_write_time_us":18761,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":454,"spinlock_wait_cycles":768} I0408 18:33:09.487347 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=2.929465 I0408 18:33:09.488535 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:09.488610 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:09.488648 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:09.488667 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 4120343 bytes I0408 18:33:09.488675 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249403341701120 or (T in {6368249403341701120})}] I0408 18:33:09.489504 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:09.519609 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:09.520180 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:09.520254 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249403470336000 or (T in {6368249403470336000})}] I0408 18:33:09.527573 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 35700 rows (1010453 bytes) I0408 18:33:09.527714 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.040s user 0.032s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":96,"data dirs.run_wall_time_us":1111,"lbm_read_time_us":30,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2691,"lbm_writes_lt_1ms":61} I0408 18:33:09.527810 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989957 I0408 18:33:09.527911 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:09.527928 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:09.527935 6439 compaction.cc:917] RowSet(28)(current size on disk: ~14370113 bytes) I0408 18:33:09.527940 6439 compaction.cc:917] RowSet(29)(current size on disk: ~1362337 bytes) I0408 18:33:09.527947 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249403504328704}] I0408 18:33:09.528074 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:09.813657 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:09.813704 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:09.813712 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249404670398464 or (T in {6368249404670398464})}] I0408 18:33:09.897403 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 405800 rows (11299855 bytes) I0408 18:33:09.897640 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.370s user 0.329s sys 0.035s Metrics: {"cfile_cache_hit":19,"cfile_cache_hit_bytes":1814531,"cfile_cache_miss":487,"cfile_cache_miss_bytes":27688009,"cfile_init":7,"data dirs.queue_time_us":19,"data dirs.run_cpu_time_us":113,"data dirs.run_wall_time_us":61216,"delta_iterators_relevant":4,"lbm_read_time_us":11509,"lbm_reads_lt_1ms":515,"lbm_write_time_us":21042,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":497,"spinlock_wait_cycles":384} I0408 18:33:09.897850 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=3.921653 I0408 18:33:09.898341 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:09.898399 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:09.898435 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:09.898469 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 5160727 bytes I0408 18:33:09.898515 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249405018927104 or (T in {6368249405018927104})}] I0408 18:33:09.898682 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:09.932608 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:09.932677 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:09.932689 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249405162516480 or (T in {6368249405162516480})}] I0408 18:33:09.938932 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 41350 rows (1173695 bytes) I0408 18:33:09.939095 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.041s user 0.035s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":54,"data dirs.run_cpu_time_us":83,"data dirs.run_wall_time_us":3049,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2245,"lbm_writes_lt_1ms":68} I0408 18:33:09.939306 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989999 I0408 18:33:09.939442 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:09.939502 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:09.939680 6439 compaction.cc:917] RowSet(30)(current size on disk: ~15761907 bytes) I0408 18:33:09.939738 6439 compaction.cc:917] RowSet(31)(current size on disk: ~1584755 bytes) I0408 18:33:09.939779 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249405192531968}] I0408 18:33:09.939898 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:10.275573 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:10.275625 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:10.275635 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249406566715392 or (T in {6368249406566715392})}] I0408 18:33:10.350409 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 447150 rows (12442667 bytes) I0408 18:33:10.350533 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.411s user 0.345s sys 0.048s Metrics: {"cfile_cache_hit":18,"cfile_cache_hit_bytes":1775165,"cfile_cache_miss":540,"cfile_cache_miss_bytes":30749815,"cfile_init":7,"data dirs.queue_time_us":53,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":52625,"delta_iterators_relevant":4,"lbm_read_time_us":11672,"lbm_reads_lt_1ms":568,"lbm_write_time_us":23625,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":545,"spinlock_wait_cycles":130688} I0408 18:33:10.350633 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=3.921653 I0408 18:33:10.350718 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:10.350733 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:10.350739 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:10.350744 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 5160727 bytes I0408 18:33:10.350754 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249406869491712 or (T in {6368249406869491712})}] I0408 18:33:10.350875 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:10.386013 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:10.386157 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:10.386178 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249407019401216 or (T in {6368249407019401216})}] I0408 18:33:10.392383 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 43000 rows (1219276 bytes) I0408 18:33:10.392532 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.042s user 0.037s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":39,"data dirs.run_cpu_time_us":76,"data dirs.run_wall_time_us":1599,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3478,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":69} I0408 18:33:10.392650 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989865 I0408 18:33:10.392745 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:10.392783 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:10.392802 6439 compaction.cc:917] RowSet(32)(current size on disk: ~17369218 bytes) I0408 18:33:10.392827 6439 compaction.cc:917] RowSet(33)(current size on disk: ~1651161 bytes) I0408 18:33:10.392859 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249407046447104 or (T in {6368249407046447104})}] I0408 18:33:10.392999 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:10.443351 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:10.444957 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000003 I0408 18:33:10.732702 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:10.732753 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:10.732764 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249408439595008 or (T in {6368249408439595008})}] I0408 18:33:10.818778 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 490150 rows (13630933 bytes) I0408 18:33:10.819922 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.427s user 0.379s sys 0.044s Metrics: {"cfile_cache_hit":19,"cfile_cache_hit_bytes":1931347,"cfile_cache_miss":587,"cfile_cache_miss_bytes":33735125,"cfile_init":7,"data dirs.queue_time_us":809,"data dirs.run_cpu_time_us":320,"data dirs.run_wall_time_us":77680,"delta_iterators_relevant":4,"lbm_read_time_us":13201,"lbm_reads_lt_1ms":615,"lbm_write_time_us":24848,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":595,"spinlock_wait_cycles":384} I0408 18:33:10.820117 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 5417040 bytes on disk I0408 18:33:10.820253 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0408 18:33:10.820336 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=3.921653 I0408 18:33:10.820411 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:10.820428 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:10.820435 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:10.820438 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 5160727 bytes I0408 18:33:10.820447 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249408794566656 or (T in {6368249408794566656})}] I0408 18:33:10.820534 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:10.856719 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:10.856827 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:10.856848 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249408945680384 or (T in {6368249408945680384})}] I0408 18:33:10.862721 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 46300 rows (1310532 bytes) I0408 18:33:10.862910 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.042s user 0.034s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":34,"data dirs.run_cpu_time_us":79,"data dirs.run_wall_time_us":1760,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2812,"lbm_writes_lt_1ms":74} I0408 18:33:10.863127 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8388558 bytes of WAL I0408 18:33:10.863314 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:10.863368 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000002 (ops 5004-9992) I0408 18:33:10.864897 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {} I0408 18:33:10.864969 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 461991 bytes on disk I0408 18:33:10.865092 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4} I0408 18:33:10.865248 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989973 I0408 18:33:10.865336 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:10.865393 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:10.865430 6439 compaction.cc:917] RowSet(34)(current size on disk: ~19047973 bytes) I0408 18:33:10.865463 6439 compaction.cc:917] RowSet(35)(current size on disk: ~1772523 bytes) I0408 18:33:10.865499 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249408976838656 or (T in {6368249408976838656})}] I0408 18:33:10.865667 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:11.302158 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:11.302217 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:11.302227 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249410772406272}] I0408 18:33:11.402221 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 536450 rows (14910033 bytes) I0408 18:33:11.402417 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.537s user 0.426s sys 0.040s Metrics: {"cfile_cache_hit":21,"cfile_cache_hit_bytes":2249993,"cfile_cache_miss":641,"cfile_cache_miss_bytes":36794245,"cfile_init":5,"data dirs.queue_time_us":18,"data dirs.run_cpu_time_us":92,"data dirs.run_wall_time_us":24611,"delta_iterators_relevant":4,"lbm_read_time_us":15382,"lbm_reads_lt_1ms":661,"lbm_write_time_us":25594,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":651,"spinlock_wait_cycles":1152} I0408 18:33:11.402606 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 5940284 bytes on disk I0408 18:33:11.402753 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} I0408 18:33:11.402853 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=4.913840 I0408 18:33:11.402921 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:11.402969 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:11.403002 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:11.403040 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 6201111 bytes I0408 18:33:11.403077 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249411185623040}] I0408 18:33:11.403236 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:11.448801 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:11.448935 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:11.448962 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249411368407040 or (T in {6368249411368407040})}] I0408 18:33:11.456290 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 58500 rows (1651964 bytes) I0408 18:33:11.456441 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.054s user 0.052s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":39,"data dirs.run_cpu_time_us":144,"data dirs.run_wall_time_us":3499,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3425,"lbm_writes_lt_1ms":88} I0408 18:33:11.456565 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989914 I0408 18:33:11.456835 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:11.456889 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:11.456923 6439 compaction.cc:917] RowSet(36)(current size on disk: ~20850317 bytes) I0408 18:33:11.456967 6439 compaction.cc:917] RowSet(37)(current size on disk: ~2251742 bytes) I0408 18:33:11.457007 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249411402518528}] I0408 18:33:11.457155 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:11.851805 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:11.851936 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:11.851995 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249413023096832 or (T in {6368249413023096832})}] I0408 18:33:11.958755 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 594950 rows (16524837 bytes) I0408 18:33:11.958956 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.502s user 0.452s sys 0.048s Metrics: {"cfile_cache_hit":21,"cfile_cache_hit_bytes":2271154,"cfile_cache_miss":711,"cfile_cache_miss_bytes":41056084,"cfile_init":6,"data dirs.queue_time_us":32,"data dirs.run_cpu_time_us":83,"data dirs.run_wall_time_us":99124,"delta_iterators_relevant":4,"lbm_read_time_us":15332,"lbm_reads_lt_1ms":735,"lbm_write_time_us":28977,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":720,"spinlock_wait_cycles":718208} I0408 18:33:11.959492 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=4.913840 I0408 18:33:11.959827 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:11.959882 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:11.959919 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:11.959955 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 6201111 bytes I0408 18:33:11.959995 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249413463560192 or (T in {6368249413463560192})}] I0408 18:33:11.960136 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:12.005440 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:12.005681 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:12.005708 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249413652193280 or (T in {6368249413652193280})}] I0408 18:33:12.014714 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 53450 rows (1511898 bytes) I0408 18:33:12.015259 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.056s user 0.045s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":32,"data dirs.run_cpu_time_us":134,"data dirs.run_wall_time_us":4267,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3656,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":80} I0408 18:33:12.015389 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989951 I0408 18:33:12.015453 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:12.015494 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:12.015516 6439 compaction.cc:917] RowSet(38)(current size on disk: ~23131646 bytes) I0408 18:33:12.015568 6439 compaction.cc:917] RowSet(39)(current size on disk: ~2047917 bytes) I0408 18:33:12.015740 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249413693632512}] I0408 18:33:12.015910 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:12.469310 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:12.469352 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:12.469358 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249415543136256 or (T in {6368249415543136256})}] I0408 18:33:12.582828 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 648400 rows (18003357 bytes) I0408 18:33:12.582911 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.567s user 0.505s sys 0.052s Metrics: {"cfile_cache_hit":18,"cfile_cache_hit_bytes":1777470,"cfile_cache_miss":778,"cfile_cache_miss_bytes":45449364,"cfile_init":7,"data dirs.queue_time_us":35,"data dirs.run_cpu_time_us":80,"data dirs.run_wall_time_us":102639,"delta_iterators_relevant":4,"lbm_read_time_us":17964,"lbm_reads_lt_1ms":806,"lbm_write_time_us":32583,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":782,"spinlock_wait_cycles":768} I0408 18:33:12.583014 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 7210808 bytes on disk I0408 18:33:12.583191 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} I0408 18:33:12.583292 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=4.913840 I0408 18:33:12.583380 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:12.583422 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:12.583451 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:12.583484 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 6201111 bytes I0408 18:33:12.583521 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249416018345984 or (T in {6368249416018345984})}] I0408 18:33:12.583667 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:12.632514 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:12.632994 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:12.633057 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249416221917184 or (T in {6368249416221917184})}] I0408 18:33:12.641175 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 59750 rows (1683607 bytes) I0408 18:33:12.641283 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.058s user 0.053s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":23,"data dirs.run_cpu_time_us":78,"data dirs.run_wall_time_us":943,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4669,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":87} I0408 18:33:12.641470 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989998 I0408 18:33:12.641588 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:12.641644 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:12.641685 6439 compaction.cc:917] RowSet(40)(current size on disk: ~25214165 bytes) I0408 18:33:12.641705 6439 compaction.cc:917] RowSet(41)(current size on disk: ~2292083 bytes) I0408 18:33:12.641714 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249416257576960 or (T in {6368249416257576960})}] I0408 18:33:12.641816 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:13.022528 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:13.023717 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000004 I0408 18:33:13.226125 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:13.226233 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:13.226258 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249418653020160 or (T in {6368249418653020160})}] I0408 18:33:13.348984 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 708150 rows (19648213 bytes) I0408 18:33:13.353868 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.712s user 0.542s sys 0.060s Metrics: {"cfile_cache_hit":17,"cfile_cache_hit_bytes":1617112,"cfile_cache_miss":847,"cfile_cache_miss_bytes":49974758,"cfile_init":6,"data dirs.queue_time_us":157,"data dirs.run_cpu_time_us":4719,"data dirs.run_wall_time_us":16614,"delta_iterators_relevant":4,"lbm_read_time_us":18561,"lbm_reads_lt_1ms":871,"lbm_write_time_us":34282,"lbm_writes_lt_1ms":856,"spinlock_wait_cycles":505856,"thread_start_us":51,"threads_started":1} I0408 18:33:13.354053 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=6.898215 I0408 18:33:13.354133 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:13.354176 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:13.354197 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:13.354233 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 8281879 bytes I0408 18:33:13.354279 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249419178139648}] I0408 18:33:13.354409 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:13.412842 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:13.412981 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:13.413043 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249419415982080 or (T in {6368249419415982080})}] I0408 18:33:13.424132 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 74350 rows (2095061 bytes) I0408 18:33:13.424484 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.070s user 0.069s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":39,"data dirs.run_cpu_time_us":86,"data dirs.run_wall_time_us":7046,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3311,"lbm_writes_lt_1ms":107} I0408 18:33:13.424671 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8389947 bytes of WAL I0408 18:33:13.424764 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:13.424787 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000003 (ops 9993-14982) I0408 18:33:13.426656 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {} I0408 18:33:13.426745 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989990 I0408 18:33:13.426901 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:13.426923 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:13.426929 6439 compaction.cc:917] RowSet(42)(current size on disk: ~27538431 bytes) I0408 18:33:13.426934 6439 compaction.cc:917] RowSet(43)(current size on disk: ~2863974 bytes) I0408 18:33:13.426942 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249419475554304}] I0408 18:33:13.427048 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:14.101192 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:14.101619 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:14.101637 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249422234591232 or (T in {6368249422237130752,6368249422234591232})}] I0408 18:33:14.242413 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 782500 rows (21695026 bytes) I0408 18:33:14.248011 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.821s user 0.598s sys 0.087s Metrics: {"cfile_cache_hit":21,"cfile_cache_hit_bytes":2259367,"cfile_cache_miss":935,"cfile_cache_miss_bytes":54762305,"cfile_init":7,"data dirs.queue_time_us":172,"data dirs.run_cpu_time_us":5318,"data dirs.run_wall_time_us":15399,"delta_iterators_relevant":4,"lbm_read_time_us":20922,"lbm_reads_lt_1ms":963,"lbm_write_time_us":38017,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":941,"spinlock_wait_cycles":1536,"thread_start_us":54,"threads_started":1} I0408 18:33:14.248234 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=7.890403 I0408 18:33:14.248351 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:14.248404 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:14.248422 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:14.248430 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 9322263 bytes I0408 18:33:14.248437 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249422839840768}] I0408 18:33:14.248558 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:14.316378 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:14.316494 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:14.316578 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249423118565376 or (T in {6368249423118565376})}] I0408 18:33:14.331056 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 89900 rows (2529790 bytes) I0408 18:33:14.331367 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.083s user 0.078s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":21,"data dirs.run_cpu_time_us":79,"data dirs.run_wall_time_us":5596,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4046,"lbm_writes_lt_1ms":125} I0408 18:33:14.331490 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 9656893 bytes on disk I0408 18:33:14.331674 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":56,"lbm_reads_lt_1ms":8} I0408 18:33:14.331743 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989993 I0408 18:33:14.331822 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:14.331840 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:14.331846 6439 compaction.cc:917] RowSet(44)(current size on disk: ~30428920 bytes) I0408 18:33:14.331851 6439 compaction.cc:917] RowSet(45)(current size on disk: ~3452789 bytes) I0408 18:33:14.331861 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249423180001280 or (T in {6368249423180001280})}] I0408 18:33:14.331988 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:14.982746 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:14.982801 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:14.982812 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249425847066624 or (T in {6368249425847066624})}] I0408 18:33:15.132795 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 872400 rows (24171706 bytes) I0408 18:33:15.137714 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.806s user 0.698s sys 0.056s Metrics: {"cfile_cache_hit":23,"cfile_cache_hit_bytes":2361301,"cfile_cache_miss":1039,"cfile_cache_miss_bytes":61190551,"cfile_init":5,"data dirs.queue_time_us":5027,"data dirs.run_cpu_time_us":4730,"data dirs.run_wall_time_us":98848,"delta_iterators_relevant":4,"lbm_read_time_us":24272,"lbm_reads_lt_1ms":1059,"lbm_write_time_us":44193,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":1047,"spinlock_wait_cycles":4224,"thread_start_us":51,"threads_started":1} I0408 18:33:15.137913 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=7.890403 I0408 18:33:15.138236 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:15.138299 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:15.138339 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:15.138378 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 9322263 bytes I0408 18:33:15.138414 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249426486312960}] I0408 18:33:15.138695 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:15.207525 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:15.207597 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:15.207609 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249426767241216 or (T in {6368249426767241216})}] I0408 18:33:15.220232 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 86200 rows (2426395 bytes) I0408 18:33:15.220306 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.082s user 0.071s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":109,"data dirs.run_wall_time_us":4534,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5224,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":120} I0408 18:33:15.220428 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989983 I0408 18:33:15.220556 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:15.220611 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:15.220625 6439 compaction.cc:917] RowSet(46)(current size on disk: ~33926018 bytes) I0408 18:33:15.220631 6439 compaction.cc:917] RowSet(47)(current size on disk: ~3305588 bytes) I0408 18:33:15.220639 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249426820120576}] I0408 18:33:15.220744 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:15.538925 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:15.539691 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000005 I0408 18:33:15.919792 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:15.919909 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:15.919966 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249429684539392 or (T in {6368249429684539392})}] I0408 18:33:16.089681 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 958600 rows (26538235 bytes) I0408 18:33:16.089771 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.869s user 0.738s sys 0.079s Metrics: {"cfile_cache_hit":25,"cfile_cache_hit_bytes":2651066,"cfile_cache_miss":1145,"cfile_cache_miss_bytes":67180390,"cfile_init":7,"data dirs.queue_time_us":152,"data dirs.run_cpu_time_us":94,"data dirs.run_wall_time_us":96743,"delta_iterators_relevant":4,"lbm_read_time_us":25769,"lbm_reads_lt_1ms":1173,"lbm_write_time_us":47161,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":1150,"spinlock_wait_cycles":351104,"thread_start_us":61,"threads_started":1} I0408 18:33:16.089901 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=8.882590 I0408 18:33:16.089988 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:16.090047 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:16.090082 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:16.090122 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 10362647 bytes I0408 18:33:16.090167 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249430381731840}] I0408 18:33:16.090308 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:16.164598 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:16.164647 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:16.164654 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249430685454336 or (T in {6368249430685454336})}] I0408 18:33:16.179615 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 93050 rows (2618814 bytes) I0408 18:33:16.179847 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.090s user 0.079s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":164,"data dirs.run_wall_time_us":4779,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5385,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":127} I0408 18:33:16.179970 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8389050 bytes of WAL I0408 18:33:16.180021 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:16.180028 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000004 (ops 14983-19969) I0408 18:33:16.181892 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {} I0408 18:33:16.181987 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.990000 I0408 18:33:16.182075 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:16.182123 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:16.182159 6439 compaction.cc:917] RowSet(48)(current size on disk: ~37276602 bytes) I0408 18:33:16.182198 6439 compaction.cc:917] RowSet(49)(current size on disk: ~3577334 bytes) I0408 18:33:16.182231 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249430760144896}] I0408 18:33:16.182390 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:16.956756 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:16.956800 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:16.956807 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249433927118848 or (T in {6368249433927118848})}] I0408 18:33:17.148176 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 1051650 rows (29099336 bytes) I0408 18:33:17.148262 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.966s user 0.823s sys 0.064s Metrics: {"cfile_cache_hit":26,"cfile_cache_hit_bytes":2711890,"cfile_cache_miss":1252,"cfile_cache_miss_bytes":73919214,"cfile_init":7,"data dirs.queue_time_us":90,"data dirs.run_cpu_time_us":99,"data dirs.run_wall_time_us":69197,"delta_iterators_relevant":4,"lbm_read_time_us":28955,"lbm_reads_lt_1ms":1280,"lbm_write_time_us":47787,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":1263,"spinlock_wait_cycles":3202304,"thread_start_us":51,"threads_started":1} I0408 18:33:17.148481 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=8.882590 I0408 18:33:17.148609 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:17.148665 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:17.149294 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:17.149343 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 10362647 bytes I0408 18:33:17.149386 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249434723164160}] I0408 18:33:17.149526 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:17.234025 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:17.234086 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:17.234107 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249435064864768 or (T in {6368249435064864768})}] I0408 18:33:17.250492 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 102150 rows (2869896 bytes) I0408 18:33:17.250586 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.102s user 0.088s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":37,"data dirs.run_cpu_time_us":99,"data dirs.run_wall_time_us":4618,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7399,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":138} I0408 18:33:17.250723 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989985 I0408 18:33:17.251272 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:17.251293 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:17.251302 6439 compaction.cc:917] RowSet(50)(current size on disk: ~40896717 bytes) I0408 18:33:17.251307 6439 compaction.cc:917] RowSet(51)(current size on disk: ~3931239 bytes) I0408 18:33:17.251315 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249435137662976 or (T in {6368249435137662976})}] I0408 18:33:17.251426 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:18.108989 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:18.109673 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000006 I0408 18:33:18.230742 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:18.230796 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:18.230808 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249439150542848 or (T in {6368249439150542848})}] I0408 18:33:18.432456 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 1153800 rows (31903588 bytes) I0408 18:33:18.436959 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 1.186s user 0.895s sys 0.076s Metrics: {"cfile_cache_hit":25,"cfile_cache_hit_bytes":2708838,"cfile_cache_miss":1379,"cfile_cache_miss_bytes":81384100,"cfile_init":7,"data dirs.queue_time_us":228,"data dirs.run_cpu_time_us":4331,"data dirs.run_wall_time_us":8323,"delta_iterators_relevant":4,"lbm_read_time_us":30906,"lbm_reads_lt_1ms":1407,"lbm_write_time_us":56868,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":1380,"spinlock_wait_cycles":10112,"thread_start_us":77,"threads_started":1} I0408 18:33:18.437054 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 12959664 bytes on disk I0408 18:33:18.437197 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0408 18:33:18.437466 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=11.859153 I0408 18:33:18.437842 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:18.437896 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:18.437935 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:18.437974 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 13483799 bytes I0408 18:33:18.438020 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249439998943232 or (T in {6368249439998943232})}] I0408 18:33:18.442494 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:18.537678 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:18.537807 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:18.537870 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249440408739840 or (T in {6368249440408739840})}] I0408 18:33:18.559667 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 125500 rows (3523782 bytes) I0408 18:33:18.560056 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.122s user 0.110s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":17,"data dirs.run_cpu_time_us":190,"data dirs.run_wall_time_us":6185,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9920,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":165} I0408 18:33:18.560158 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8388966 bytes of WAL I0408 18:33:18.560209 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:18.560238 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000005 (ops 19970-24955) I0408 18:33:18.562135 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {} I0408 18:33:18.562258 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989993 I0408 18:33:18.562347 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:18.562369 6439 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:18.562376 6439 compaction.cc:917] RowSet(52)(current size on disk: ~44863252 bytes) I0408 18:33:18.562381 6439 compaction.cc:917] RowSet(53)(current size on disk: ~4835583 bytes) I0408 18:33:18.562391 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249440509227008 or (T in {6368249440509227008})}] I0408 18:33:18.562506 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:19.579639 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:19.607872 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:19.608253 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:19.608281 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249444792209408 or (T in {6368249444792209408})}] I0408 18:33:19.833240 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 1279300 rows (35357034 bytes) I0408 18:33:19.833981 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 1.272s user 0.997s sys 0.092s Metrics: {"cfile_cache_hit":27,"cfile_cache_hit_bytes":3053865,"cfile_cache_miss":1523,"cfile_cache_miss_bytes":90171067,"cfile_init":7,"data dirs.queue_time_us":164,"data dirs.run_cpu_time_us":696,"data dirs.run_wall_time_us":25792,"delta_iterators_relevant":4,"lbm_read_time_us":34149,"lbm_reads_lt_1ms":1551,"lbm_write_time_us":60900,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":1548,"spinlock_wait_cycles":773760,"thread_start_us":53,"threads_started":1} I0408 18:33:19.835252 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=12.851340 I0408 18:33:19.836068 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:19.836122 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:19.836148 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:19.836163 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 14524183 bytes I0408 18:33:19.836171 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249445727281152}] I0408 18:33:19.837743 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:19.944649 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:19.944710 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:19.944720 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249446170927104}] I0408 18:33:19.967188 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 136600 rows (3831697 bytes) I0408 18:33:19.967499 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.132s user 0.120s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":22,"data dirs.run_cpu_time_us":73,"data dirs.run_wall_time_us":7649,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9589,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":178} I0408 18:33:19.967716 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989994 I0408 18:33:19.967833 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 3 rowsets to compact or flush I0408 18:33:19.967857 6439 compaction.cc:914] Selected 3 rowsets to compact: I0408 18:33:19.967864 6439 compaction.cc:917] RowSet(54)(current size on disk: ~48401297 bytes) I0408 18:33:19.967869 6439 compaction.cc:917] RowSet(55)(current size on disk: ~1346322 bytes) I0408 18:33:19.967873 6439 compaction.cc:917] RowSet(56)(current size on disk: ~5272173 bytes) I0408 18:33:19.967881 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249446266310656}] I0408 18:33:19.967999 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:20.682756 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:20.683616 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000007 I0408 18:33:20.946041 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:21.065425 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:21.065475 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:21.065485 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249450759413760 or (T in {6368249450759413760})}] I0408 18:33:21.313764 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 1415900 rows (39103705 bytes) I0408 18:33:21.313967 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 1.346s user 1.087s sys 0.112s Metrics: {"cfile_cache_hit":38,"cfile_cache_hit_bytes":4201674,"cfile_cache_miss":1688,"cfile_cache_miss_bytes":99005098,"cfile_init":11,"data dirs.queue_time_us":224,"data dirs.run_cpu_time_us":236,"data dirs.run_wall_time_us":57632,"delta_iterators_relevant":6,"lbm_read_time_us":35763,"lbm_reads_lt_1ms":1732,"lbm_write_time_us":67730,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":1705,"spinlock_wait_cycles":466176,"thread_start_us":49,"threads_started":1} I0408 18:33:21.319016 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=12.851340 I0408 18:33:21.319365 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:21.319386 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:21.319392 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:21.319397 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 14524183 bytes I0408 18:33:21.319406 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249451781455872 or (T in {6368249451781455872})}] I0408 18:33:21.321398 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:21.432394 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:21.432608 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:21.432683 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249452266295296 or (T in {6368249452266295296})}] I0408 18:33:21.457140 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 142800 rows (4003219 bytes) I0408 18:33:21.457417 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.138s user 0.127s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":22,"data dirs.run_cpu_time_us":77,"data dirs.run_wall_time_us":5605,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8243,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":187} I0408 18:33:21.457522 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8388847 bytes of WAL I0408 18:33:21.457617 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:21.457657 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000006 (ops 24956-29941) I0408 18:33:21.459437 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.002s Metrics: {} I0408 18:33:21.459555 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 17463301 bytes on disk I0408 18:33:21.459710 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":37,"lbm_reads_lt_1ms":12} I0408 18:33:21.459800 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989987 I0408 18:33:21.460171 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 3 rowsets to compact or flush I0408 18:33:21.460285 6439 compaction.cc:914] Selected 3 rowsets to compact: I0408 18:33:21.460304 6439 compaction.cc:917] RowSet(57)(current size on disk: ~48393152 bytes) I0408 18:33:21.460309 6439 compaction.cc:917] RowSet(58)(current size on disk: ~6660539 bytes) I0408 18:33:21.460314 6439 compaction.cc:917] RowSet(59)(current size on disk: ~5516534 bytes) I0408 18:33:21.460322 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249452378501120}] I0408 18:33:21.460434 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:22.505435 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:22.716367 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:22.716599 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:22.716622 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249457524994048 or (T in {6368249457524994048})}] I0408 18:33:22.989456 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 1558700 rows (43022041 bytes) I0408 18:33:22.989603 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 1.529s user 1.233s sys 0.098s Metrics: {"cfile_cache_hit":38,"cfile_cache_hit_bytes":3901132,"cfile_cache_miss":1854,"cfile_cache_miss_bytes":109714406,"cfile_init":8,"data dirs.queue_time_us":191,"data dirs.run_cpu_time_us":163,"data dirs.run_wall_time_us":47293,"delta_iterators_relevant":6,"lbm_read_time_us":39630,"lbm_reads_lt_1ms":1886,"lbm_write_time_us":73311,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":1879,"spinlock_wait_cycles":2615552,"thread_start_us":45,"threads_started":1} I0408 18:33:22.991235 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 17576023 bytes on disk I0408 18:33:22.991411 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":21,"lbm_reads_lt_1ms":8} I0408 18:33:22.991655 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=14.835715 I0408 18:33:22.995049 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:22.995108 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:22.995172 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:22.995214 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 16604951 bytes I0408 18:33:22.995260 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249458662862848}] I0408 18:33:22.995390 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:23.128702 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:23.128804 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:23.128859 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249459214024704 or (T in {6368249459214024704})}] I0408 18:33:23.156013 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 160200 rows (4489131 bytes) I0408 18:33:23.156407 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.161s user 0.143s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":23,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":7773,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14567,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":207} I0408 18:33:23.156576 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 1705578 bytes on disk I0408 18:33:23.156728 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4} I0408 18:33:23.156786 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989997 I0408 18:33:23.156852 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 3 rowsets to compact or flush I0408 18:33:23.156869 6439 compaction.cc:914] Selected 3 rowsets to compact: I0408 18:33:23.156877 6439 compaction.cc:917] RowSet(60)(current size on disk: ~48391681 bytes) I0408 18:33:23.156882 6439 compaction.cc:917] RowSet(61)(current size on disk: ~12206383 bytes) I0408 18:33:23.156885 6439 compaction.cc:917] RowSet(62)(current size on disk: ~6194709 bytes) I0408 18:33:23.156893 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249459327561728}] I0408 18:33:23.156991 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:23.275564 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:23.276733 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000008 I0408 18:33:24.120859 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:24.580493 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:24.580552 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:24.580564 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249465150246912 or (T in {6368249465150246912})}] I0408 18:33:24.894248 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 1718900 rows (47410316 bytes) I0408 18:33:24.894639 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 1.738s user 1.322s sys 0.160s Metrics: {"cfile_cache_hit":41,"cfile_cache_hit_bytes":4464654,"cfile_cache_miss":2045,"cfile_cache_miss_bytes":120829826,"cfile_init":8,"data dirs.queue_time_us":153,"data dirs.run_cpu_time_us":366,"data dirs.run_wall_time_us":173428,"delta_iterators_relevant":6,"lbm_read_time_us":45191,"lbm_reads_lt_1ms":2077,"lbm_write_time_us":83862,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":2068,"spinlock_wait_cycles":1009536,"thread_start_us":65,"threads_started":1} I0408 18:33:24.894742 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=16.820090 I0408 18:33:24.895349 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:24.895408 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:24.895448 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:24.895489 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 18685719 bytes I0408 18:33:24.895552 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249466449301504}] I0408 18:33:24.896713 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:25.056846 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:25.057250 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:25.057279 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249467111428096 or (T in {6368249467111428096})}] I0408 18:33:25.090327 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 182550 rows (5109878 bytes) I0408 18:33:25.090822 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.196s user 0.173s sys 0.012s Metrics: {"cfile_init":1,"data dirs.queue_time_us":21,"data dirs.run_cpu_time_us":115,"data dirs.run_wall_time_us":7594,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":16333,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":232} I0408 18:33:25.090955 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8390700 bytes of WAL I0408 18:33:25.091022 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:25.091035 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000007 (ops 29942-34928) I0408 18:33:25.092953 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {} I0408 18:33:25.093052 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 21357686 bytes on disk I0408 18:33:25.093252 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":42,"lbm_reads_lt_1ms":12} I0408 18:33:25.093353 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989993 I0408 18:33:25.093462 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 3 rowsets to compact or flush I0408 18:33:25.093516 6439 compaction.cc:914] Selected 3 rowsets to compact: I0408 18:33:25.093572 6439 compaction.cc:917] RowSet(63)(current size on disk: ~48383374 bytes) I0408 18:33:25.093616 6439 compaction.cc:917] RowSet(64)(current size on disk: ~18430015 bytes) I0408 18:33:25.093648 6439 compaction.cc:917] RowSet(65)(current size on disk: ~7064491 bytes) I0408 18:33:25.093689 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249467263102976}] I0408 18:33:25.093813 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:25.873530 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:25.875764 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000009 I0408 18:33:26.082234 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:26.628407 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:26.628460 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:26.628471 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249473545388032 or (T in {6368249473545388032})}] I0408 18:33:26.959743 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 1901450 rows (52406248 bytes) I0408 18:33:26.970826 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 1.877s user 1.501s sys 0.098s Metrics: {"cfile_cache_hit":45,"cfile_cache_hit_bytes":5181739,"cfile_cache_miss":2261,"cfile_cache_miss_bytes":133401349,"cfile_init":8,"data dirs.queue_time_us":263,"data dirs.run_cpu_time_us":9944,"data dirs.run_wall_time_us":73713,"delta_iterators_relevant":6,"lbm_read_time_us":47854,"lbm_reads_lt_1ms":2293,"lbm_write_time_us":89926,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":2282,"spinlock_wait_cycles":1365248,"thread_start_us":43,"threads_started":1} I0408 18:33:26.971333 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 21487694 bytes on disk I0408 18:33:26.971505 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":27,"lbm_reads_lt_1ms":8} I0408 18:33:26.971617 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=18.804465 I0408 18:33:26.971731 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:26.971779 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:26.971817 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:26.971853 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 20766487 bytes I0408 18:33:26.971889 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249474954993664}] I0408 18:33:26.975067 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:27.136008 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:27.136487 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:27.136588 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249475627307008 or (T in {6368249475627307008})}] I0408 18:33:27.171459 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 200150 rows (5601084 bytes) I0408 18:33:27.171617 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.200s user 0.186s sys 0.007s Metrics: {"cfile_init":1,"data dirs.queue_time_us":20,"data dirs.run_cpu_time_us":79,"data dirs.run_wall_time_us":17134,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12308,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":256} I0408 18:33:27.171735 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 2148661 bytes on disk I0408 18:33:27.171891 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4} I0408 18:33:27.171946 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8393991 bytes of WAL I0408 18:33:27.172004 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:27.172060 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000008 (ops 34929-39917) I0408 18:33:27.173952 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {} I0408 18:33:27.174019 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989993 I0408 18:33:27.174105 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 3 rowsets to compact or flush I0408 18:33:27.174126 6439 compaction.cc:914] Selected 3 rowsets to compact: I0408 18:33:27.174134 6439 compaction.cc:917] RowSet(66)(current size on disk: ~48375369 bytes) I0408 18:33:27.174139 6439 compaction.cc:917] RowSet(67)(current size on disk: ~25518573 bytes) I0408 18:33:27.174144 6439 compaction.cc:917] RowSet(68)(current size on disk: ~7749745 bytes) I0408 18:33:27.174154 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249475782393856 or (T in {6368249475782393856})}] I0408 18:33:27.174300 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:28.161448 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:28.477860 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:28.478512 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000010 I0408 18:33:28.843421 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:28.843937 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:28.843958 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249482348322816 or (T in {6368249482348322816})}] I0408 18:33:29.236451 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 2101600 rows (57892159 bytes) I0408 18:33:29.236554 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 2.062s user 1.621s sys 0.166s Metrics: {"cfile_cache_hit":52,"cfile_cache_hit_bytes":5949847,"cfile_cache_miss":2498,"cfile_cache_miss_bytes":147200813,"cfile_init":8,"data dirs.queue_time_us":195,"data dirs.run_cpu_time_us":178,"data dirs.run_wall_time_us":48131,"delta_iterators_relevant":6,"lbm_read_time_us":55001,"lbm_reads_lt_1ms":2530,"lbm_write_time_us":98432,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":2518,"spinlock_wait_cycles":5699072,"thread_start_us":122,"threads_started":2} I0408 18:33:29.236747 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 23774408 bytes on disk I0408 18:33:29.237009 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":74,"lbm_reads_lt_1ms":8} I0408 18:33:29.237125 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=19.796653 I0408 18:33:29.237243 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:29.237291 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:29.237309 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:29.237315 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 21806871 bytes I0408 18:33:29.237323 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249484231430144 or (T in {6368249484231430144})}] I0408 18:33:29.239719 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:29.396034 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:29.396450 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:29.396507 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249484881440768 or (T in {6368249484884508672,6368249484882108416,6368249484885495808,6368249484881440768})}] I0408 18:33:29.433643 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 208800 rows (5842568 bytes) I0408 18:33:29.434206 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.197s user 0.173s sys 0.020s Metrics: {"cfile_init":1,"data dirs.queue_time_us":25,"data dirs.run_cpu_time_us":102,"data dirs.run_wall_time_us":18432,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14783,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":265} I0408 18:33:29.434391 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8388700 bytes of WAL I0408 18:33:29.434499 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:29.434517 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000009 (ops 39918-44903) I0408 18:33:29.436342 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.001s Metrics: {} I0408 18:33:29.436437 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=0.989995 I0408 18:33:29.436558 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 3 rowsets to compact or flush I0408 18:33:29.436581 6439 compaction.cc:914] Selected 3 rowsets to compact: I0408 18:33:29.436595 6439 compaction.cc:917] RowSet(69)(current size on disk: ~48366723 bytes) I0408 18:33:29.436602 6439 compaction.cc:917] RowSet(70)(current size on disk: ~33299844 bytes) I0408 18:33:29.436606 6439 compaction.cc:917] RowSet(71)(current size on disk: ~8091852 bytes) I0408 18:33:29.436615 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249485051392000}] I0408 18:33:29.436738 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:30.666242 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:31.129098 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:31.131525 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000011 I0408 18:33:31.471539 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:31.471586 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:31.471593 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249493379940352 or (T in {6368249493379940352})}] I0408 18:33:31.881912 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 2310400 rows (63605561 bytes) I0408 18:33:31.882004 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 2.445s user 1.810s sys 0.138s Metrics: {"cfile_cache_hit":54,"cfile_cache_hit_bytes":6245218,"cfile_cache_miss":2742,"cfile_cache_miss_bytes":162121526,"cfile_init":9,"data dirs.queue_time_us":171,"data dirs.run_cpu_time_us":189,"data dirs.run_wall_time_us":67262,"delta_iterators_relevant":6,"lbm_read_time_us":59233,"lbm_reads_lt_1ms":2778,"lbm_write_time_us":106144,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":2770,"spinlock_wait_cycles":4472832,"thread_start_us":96,"threads_started":2} I0408 18:33:31.882136 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=23.765403 I0408 18:33:31.882207 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:31.882272 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:31.882323 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:31.882362 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 25968407 bytes I0408 18:33:31.882406 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249495064834048 or (T in {6368249495064834048})}] I0408 18:33:31.886657 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:32.087020 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:32.087090 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:32.087103 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249495904698368 or (T in {6368249495904698368})}] I0408 18:33:32.129583 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 255200 rows (7128002 bytes) I0408 18:33:32.129737 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.248s user 0.226s sys 0.008s Metrics: {"cfile_init":1,"data dirs.queue_time_us":23,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":16906,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":23972,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":317} I0408 18:33:32.129890 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:32.133388 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:32.133826 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:32.134003 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:32.134023 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:32.134032 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249496097681408}] I0408 18:33:32.134409 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:32.155086 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:32.155282 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:32.155305 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249496185577472 or (T in {6368249496185577472})}] I0408 18:33:32.158433 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 22900 rows (652586 bytes) I0408 18:33:32.158500 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.028s user 0.023s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":39,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":673,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1298,"lbm_writes_lt_1ms":47} I0408 18:33:32.158653 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8393747 bytes of WAL I0408 18:33:32.158740 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:32.159092 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000010 (ops 44904-49892) I0408 18:33:32.161823 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.003s user 0.000s sys 0.002s Metrics: {} I0408 18:33:32.161890 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 29147599 bytes on disk I0408 18:33:32.162075 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":42,"lbm_reads_lt_1ms":16} I0408 18:33:32.162206 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=1.989963 I0408 18:33:32.162554 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 4 rowsets to compact or flush I0408 18:33:32.162608 6439 compaction.cc:914] Selected 4 rowsets to compact: I0408 18:33:32.162647 6439 compaction.cc:917] RowSet(72)(current size on disk: ~48361887 bytes) I0408 18:33:32.162683 6439 compaction.cc:917] RowSet(73)(current size on disk: ~41400715 bytes) I0408 18:33:32.162719 6439 compaction.cc:917] RowSet(74)(current size on disk: ~9894965 bytes) I0408 18:33:32.162762 6439 compaction.cc:917] RowSet(75)(current size on disk: ~876181 bytes) I0408 18:33:32.162811 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249496213184512 or (T in {6368249496214831104,6368249496216858624})}] I0408 18:33:32.162940 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:33.156128 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:33.794778 6578 log.cc:644] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Max segment size reached. Starting new segment allocation I0408 18:33:33.795922 6578 log.cc:571] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000012 I0408 18:33:34.140825 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:34.217053 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:34.217118 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:34.217129 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249504627630080 or (T in {6368249504627630080})}] I0408 18:33:34.699136 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction successful on 2588500 rows (71208846 bytes) I0408 18:33:34.699817 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: CompactRowSetsOp(4754065036374334a11711d7289d68ec) complete. Timing: real 2.537s user 2.069s sys 0.206s Metrics: {"cfile_cache_hit":65,"cfile_cache_hit_bytes":7261608,"cfile_cache_miss":3077,"cfile_cache_miss_bytes":181311812,"cfile_init":11,"data dirs.queue_time_us":26054,"data dirs.run_cpu_time_us":384,"data dirs.run_wall_time_us":159204,"delta_iterators_relevant":8,"lbm_read_time_us":70001,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3120,"lbm_write_time_us":127604,"lbm_writes_1-10_ms":11,"lbm_writes_lt_1ms":3106,"spinlock_wait_cycles":5727360,"thread_start_us":107,"threads_started":2} I0408 18:33:34.700407 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=22.773215 I0408 18:33:34.702594 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:34.702751 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:34.702780 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:34.702806 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 24928023 bytes I0408 18:33:34.702834 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249506620047360}] I0408 18:33:34.702955 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:34.887941 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:34.890676 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:34.890766 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249507375902720 or (T in {6368249507376369664,6368249507378401280,6368249507375902720})}] I0408 18:33:34.933089 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 239350 rows (6685358 bytes) I0408 18:33:34.933604 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.233s user 0.200s sys 0.018s Metrics: {"cfile_init":1,"data dirs.queue_time_us":1353,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":17858,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14212,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":300,"spinlock_wait_cycles":384} I0408 18:33:34.933709 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling LogGCOp(4754065036374334a11711d7289d68ec): free 8390535 bytes of WAL I0408 18:33:34.933782 6439 log_reader.cc:376] T 4754065036374334a11711d7289d68ec: removed 1 log segments from log reader I0408 18:33:34.933828 6439 log.cc:856] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Deleting log segment in path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/4754065036374334a11711d7289d68ec/wal-000000011 (ops 49893-54879) I0408 18:33:34.935755 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: LogGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.002s user 0.000s sys 0.001s Metrics: {} I0408 18:33:34.935879 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec): 31934268 bytes on disk I0408 18:33:34.936233 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: UndoDeltaBlockGCOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":45,"lbm_reads_lt_1ms":16} I0408 18:33:34.936370 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling FlushMRSOp(4754065036374334a11711d7289d68ec): perf score=1.937278 I0408 18:33:34.936693 6439 tablet.cc:1153] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:34.936712 6439 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:34.936718 6439 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:34.936724 6439 tablet.cc:1155] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Memstore in-memory size: 3079959 bytes I0408 18:33:34.936733 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249507578818560 or (T in {6368249507578818560})}] I0408 18:33:34.936900 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:34.956040 6439 tablet.cc:1529] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:34.956231 6439 tablet.cc:1584] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:34.956292 6439 tablet.cc:1586] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249507658891264 or (T in {6368249507658891264})}] I0408 18:33:34.959808 6439 tablet.cc:1634] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Flush successful on 21450 rows (610926 bytes) I0408 18:33:34.959975 6439 maintenance_manager.cc:492] P e0de2f3696ba40e092c7bb36723895ce: FlushMRSOp(4754065036374334a11711d7289d68ec) complete. Timing: real 0.023s user 0.013s sys 0.010s Metrics: {"cfile_init":1,"data dirs.queue_time_us":19,"data dirs.run_cpu_time_us":67,"data dirs.run_wall_time_us":654,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1118,"lbm_writes_lt_1ms":45} I0408 18:33:34.960692 6554 maintenance_manager.cc:302] P e0de2f3696ba40e092c7bb36723895ce: Scheduling CompactRowSetsOp(4754065036374334a11711d7289d68ec): perf score=1.989761 I0408 18:33:34.960779 6439 tablet.cc:1664] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: stage 1 complete, picked 5 rowsets to compact or flush I0408 18:33:34.960793 6439 compaction.cc:914] Selected 5 rowsets to compact: I0408 18:33:34.960798 6439 compaction.cc:917] RowSet(76)(current size on disk: ~48350669 bytes) I0408 18:33:34.960803 6439 compaction.cc:917] RowSet(77)(current size on disk: ~48355077 bytes) I0408 18:33:34.960808 6439 compaction.cc:917] RowSet(78)(current size on disk: ~3844367 bytes) I0408 18:33:34.960813 6439 compaction.cc:917] RowSet(79)(current size on disk: ~9278359 bytes) I0408 18:33:34.960816 6439 compaction.cc:917] RowSet(80)(current size on disk: ~817254 bytes) I0408 18:33:34.960824 6439 tablet.cc:1444] T 4754065036374334a11711d7289d68ec P e0de2f3696ba40e092c7bb36723895ce: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249507674861568 or (T in {6368249507674861568})}] I0408 18:33:34.960945 6439 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_recovery-itest.cc:202: Failure Value of: !orphaned_block_ids.empty() Actual: false Expected: true /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:308: Failure Failed Timed out waiting for assertion to pass. I0408 18:33:35.667907 6360 external_mini_cluster-itest-base.cc:80] Found fatal failure I0408 18:33:35.667963 6360 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID e0de2f3696ba40e092c7bb36723895ce and pid 6426 ************************ BEGIN STACKS ************************** 0x00007fdce5e2bb9d in ?? () Id Target Id Frame * 1 process 6426 "kudu-tserver" 0x00007fdce5e2bb9d in ?? () Thread 1 (process 6426): #0 0x00007fdce5e2bb9d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0408 18:33:35.858587 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 6426 W0408 18:33:35.911600 6375 connection.cc:513] client connection to 127.6.54.1:33437 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:35.911676 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 6366 W0408 18:33:35.911600 6376 connection.cc:513] server connection from 127.6.54.1:60759 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:35.914870 6365 connection.cc:513] client connection to 127.6.54.62:42051 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:35.914866 6561 connection.cc:513] client connection to 127.6.54.62:42051 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:35.915536 6360 test_util.cc:123] ----------------------------------------------- I0408 18:33:35.915554 6360 test_util.cc:124] Had fatal failures, leaving test files at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks_1.1554748385377146-6360 [ FAILED ] BlockManagerType/TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks/1, where GetParam() = "log" (30530 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestRestartWithOrphanedReplicates/0 I0408 18:33:35.916363 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data --block_manager=file --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:35.951426 6627 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:35.951588 6627 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:35.952373 6627 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:35.952389 6627 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:35.952392 6627 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:35.953472 6627 master_main.cc:66] Master server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.6627 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:35.954345 6627 master_main.cc:73] Initializing master server... I0408 18:33:35.954553 6627 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 26511us I0408 18:33:35.954782 6627 fs_manager.cc:260] Metadata directory not provided I0408 18:33:35.954823 6627 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:35.954906 6627 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:35.954924 6627 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:35.956331 6627 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "4d02e946a9494cd4bc7e2c3035255a97" format_stamp: "Formatted at 2019-04-08 18:33:35 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:35.956511 6627 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "4d02e946a9494cd4bc7e2c3035255a97" format_stamp: "Formatted at 2019-04-08 18:33:35 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:35.957765 6627 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0408 18:33:35.958015 6627 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:35.958050 6627 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:35.958151 6627 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:35.958185 6627 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/wal uuid: "4d02e946a9494cd4bc7e2c3035255a97" format_stamp: "Formatted at 2019-04-08 18:33:35 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:35.958225 6627 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:35.973300 6627 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:35.973390 6627 master_main.cc:76] Starting Master server... I0408 18:33:35.973518 6642 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:35.973554 6642 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:35.973562 6642 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:35.976011 6627 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:36049 I0408 18:33:35.976071 6627 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:35.976104 6627 webserver.cc:184] Document root disabled I0408 18:33:35.976302 6627 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:38567/ I0408 18:33:35.976562 6627 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:35.977010 6686 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:35.978893 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 6627 I0408 18:33:35.979113 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36049 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} I0408 18:33:35.979444 6686 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97: Bootstrap starting. I0408 18:33:35.980146 6686 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97: No blocks or log segments found. Creating new log. I0408 18:33:35.980419 6686 log.cc:520] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97: Log is configured to *not* fsync() on all Append() calls I0408 18:33:35.980942 6686 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97: No bootstrap required, opened a new log I0408 18:33:35.982707 6686 raft_consensus.cc:304] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e946a9494cd4bc7e2c3035255a97" member_type: VOTER } I0408 18:33:35.982759 6686 raft_consensus.cc:330] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:35.982782 6686 raft_consensus.cc:605] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d02e946a9494cd4bc7e2c3035255a97, State: Initialized, Role: FOLLOWER I0408 18:33:35.982916 6686 consensus_queue.cc:226] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e946a9494cd4bc7e2c3035255a97" member_type: VOTER } I0408 18:33:35.982945 6686 raft_consensus.cc:344] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:35.982970 6686 raft_consensus.cc:436] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:35.983006 6686 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97: Snoozing failure detection for 1.803s (starting election) I0408 18:33:35.983031 6686 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:35.983652 6686 raft_consensus.cc:458] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e946a9494cd4bc7e2c3035255a97" member_type: VOTER } I0408 18:33:35.983721 6686 leader_election.cc:258] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:35.983868 6689 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97: Snoozing failure detection for 2.090s (election complete) I0408 18:33:35.983929 6689 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:35.984022 6689 raft_consensus.cc:572] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [term 1 LEADER]: Becoming Leader. State: Replica: 4d02e946a9494cd4bc7e2c3035255a97, State: Running, Role: LEADER I0408 18:33:35.984170 6689 consensus_queue.cc:203] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e946a9494cd4bc7e2c3035255a97" member_type: VOTER } I0408 18:33:35.984443 6691 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d02e946a9494cd4bc7e2c3035255a97. Latest consensus state: current_term: 1 leader_uuid: "4d02e946a9494cd4bc7e2c3035255a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e946a9494cd4bc7e2c3035255a97" member_type: VOTER } } I0408 18:33:35.984443 6686 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "4d02e946a9494cd4bc7e2c3035255a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e946a9494cd4bc7e2c3035255a97" member_type: VOTER } } I0408 18:33:35.984627 6686 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [sys.catalog]: This master's current role is: LEADER I0408 18:33:35.984551 6691 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [sys.catalog]: This master's current role is: LEADER I0408 18:33:35.984499 6690 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d02e946a9494cd4bc7e2c3035255a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e946a9494cd4bc7e2c3035255a97" member_type: VOTER } } I0408 18:33:35.984815 6690 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [sys.catalog]: This master's current role is: LEADER I0408 18:33:35.984833 6686 sys_catalog.cc:422] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:35.984972 6693 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:35.985090 6627 master_main.cc:79] Master server successfully started. I0408 18:33:35.985525 6693 catalog_manager.cc:983] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:36.013370 6687 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:36.013525 6687 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:36.014222 6687 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:36.014252 6687 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:36.015249 6687 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36049 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.6687 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:36.016261 6687 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:36.016505 6687 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 27011us I0408 18:33:36.016659 6687 fs_manager.cc:260] Metadata directory not provided I0408 18:33:36.016692 6687 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:36.016765 6687 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:36.016783 6687 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:36.018049 6687 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "a8cbf6f597154071980a76fd950b98ca" format_stamp: "Formatted at 2019-04-08 18:33:36 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:36.018225 6687 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "a8cbf6f597154071980a76fd950b98ca" format_stamp: "Formatted at 2019-04-08 18:33:36 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:36.019490 6687 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s I0408 18:33:36.019870 6687 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:36.019909 6687 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:36.020048 6687 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:36.020094 6687 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "a8cbf6f597154071980a76fd950b98ca" format_stamp: "Formatted at 2019-04-08 18:33:36 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:36.020139 6687 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:36.031898 6693 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:36.032543 6693 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:36.036772 6693 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 4d02e946a9494cd4bc7e2c3035255a97: Generated new TSK 0 I0408 18:33:36.101886 6687 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:36.102197 6687 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:36.102283 6687 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:36.106806 6687 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:43433 I0408 18:33:36.106899 6687 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:36.106918 6687 webserver.cc:184] Document root disabled I0408 18:33:36.108263 6687 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:41205/ I0408 18:33:36.108546 6687 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:36.109381 6687 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:36.108989 6817 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:36.109428 6817 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:36.109436 6817 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:36.113013 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 6687 I0408 18:33:36.113787 6816 heartbeater.cc:346] Connected to a master server at 127.6.54.62:36049 I0408 18:33:36.113826 6816 heartbeater.cc:426] Registering TS with master... I0408 18:33:36.113921 6816 heartbeater.cc:475] Master 127.6.54.62:36049 requested a full tablet report, sending... I0408 18:33:36.114404 6653 ts_manager.cc:81] Registered new tserver with Master: a8cbf6f597154071980a76fd950b98ca (127.6.54.1:43433) I0408 18:33:36.115316 6653 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:55971 I0408 18:33:36.116060 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:36.120453 6360 test_util.cc:200] Using random seed: 319739525 I0408 18:33:36.125321 6653 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39490: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:36.130136 6753 tablet_service.cc:758] Processing CreateTablet for tablet 2248fdd624374521b5944813156ec928 (table=test-workload [id=f0d9e583fc834ba49dd4460aafecda09]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:36.130342 6753 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 2248fdd624374521b5944813156ec928. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:36.132799 6753 ts_tablet_manager.cc:1080] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:36.133152 6833 ts_tablet_manager.cc:938] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Bootstrapping tablet I0408 18:33:36.133330 6833 tablet_bootstrap.cc:436] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Bootstrap starting. I0408 18:33:36.133924 6833 tablet_bootstrap.cc:581] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: No blocks or log segments found. Creating new log. I0408 18:33:36.134215 6833 log.cc:520] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Log is configured to *not* fsync() on all Append() calls I0408 18:33:36.135567 6833 tablet_bootstrap.cc:436] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: No bootstrap required, opened a new log I0408 18:33:36.135637 6833 ts_tablet_manager.cc:955] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0408 18:33:36.137030 6833 raft_consensus.cc:304] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } } I0408 18:33:36.137071 6833 raft_consensus.cc:330] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:36.137089 6833 raft_consensus.cc:605] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cbf6f597154071980a76fd950b98ca, State: Initialized, Role: FOLLOWER I0408 18:33:36.137198 6833 consensus_queue.cc:226] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } } I0408 18:33:36.137243 6833 raft_consensus.cc:344] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:36.137257 6833 raft_consensus.cc:436] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:36.137284 6833 raft_consensus.cc:2700] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Snoozing failure detection for 2.069s (starting election) I0408 18:33:36.137303 6833 raft_consensus.cc:2752] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:36.137883 6833 raft_consensus.cc:458] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } } I0408 18:33:36.137948 6833 leader_election.cc:258] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:36.138141 6833 ts_tablet_manager.cc:983] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:36.138145 6816 heartbeater.cc:467] Master 127.6.54.62:36049 was elected leader, sending a full tablet report... I0408 18:33:36.143219 6835 raft_consensus.cc:2700] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Snoozing failure detection for 1.608s (election complete) I0408 18:33:36.143246 6835 raft_consensus.cc:2507] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:36.143319 6835 raft_consensus.cc:572] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 1 LEADER]: Becoming Leader. State: Replica: a8cbf6f597154071980a76fd950b98ca, State: Running, Role: LEADER I0408 18:33:36.143391 6835 consensus_queue.cc:203] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } } I0408 18:33:36.143988 6653 catalog_manager.cc:3595] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca reported cstate change: term changed from 0 to 1, leader changed from to a8cbf6f597154071980a76fd950b98ca (127.6.54.1). New cstate: current_term: 1 leader_uuid: "a8cbf6f597154071980a76fd950b98ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } health_report { overall_health: HEALTHY } } } I0408 18:33:36.149261 6813 generic_service.cc:146] {username='slave'} at 127.0.0.1:50016 changed flags via RPC: fault_crash_before_append_commit from '0' to '0.05' W0408 18:33:36.161109 6845 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:36.161134 6845 fault_injection.cc:42] THIS SERVER MAY CRASH! E0408 18:33:36.169137 6845 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_append_commit (process will exit) W0408 18:33:36.172660 6826 connection.cc:513] client connection to 127.6.54.1:43433 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:36.172699 6626 connection.cc:513] client connection to 127.6.54.1:43433 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:36.172701 6636 connection.cc:513] server connection from 127.6.54.1:55971 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:36.172746 6826 meta_cache.cc:219] Tablet 2248fdd624374521b5944813156ec928: Replica a8cbf6f597154071980a76fd950b98ca (127.6.54.1:43433) has failed: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:36.172904 6826 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0408 18:33:36.172904 6635 connection.cc:513] client connection to 127.6.54.1:43433 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:37.210063 6826 batcher.cc:349] Timed out: Failed to write batch of 50 ops to tablet 2248fdd624374521b5944813156ec928 after 42 attempt(s): Failed to write to server: (no server available): Write(tablet: 2248fdd624374521b5944813156ec928, num_ops: 50, num_attempts: 42) passed its deadline: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:37.215868 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:43433 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=41205 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:36049 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:37.248818 6849 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:37.248956 6849 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:37.249631 6849 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:37.249655 6849 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:37.250586 6849 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:43433 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=41205 --tserver_master_addrs=127.6.54.62:36049 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.6849 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:37.251576 6849 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:37.251811 6849 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 27511us I0408 18:33:37.251948 6849 fs_manager.cc:260] Metadata directory not provided I0408 18:33:37.251987 6849 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:37.252952 6849 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:37.253156 6849 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:37.253182 6849 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:37.253271 6849 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:37.253644 6849 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "a8cbf6f597154071980a76fd950b98ca" format_stamp: "Formatted at 2019-04-08 18:33:36 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:37.253685 6849 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:37.302347 6849 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:37.302598 6849 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:37.302618 6849 ts_tablet_manager.cc:915] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Loading tablet metadata I0408 18:33:37.303239 6849 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:37.303761 6849 ts_tablet_manager.cc:1080] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:37.303903 6849 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:37.303905 6864 ts_tablet_manager.cc:938] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Bootstrapping tablet I0408 18:33:37.304049 6864 tablet_bootstrap.cc:436] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Bootstrap starting. I0408 18:33:37.309160 6849 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:43433 I0408 18:33:37.309247 6849 webserver.cc:173] Starting webserver on 127.6.54.1:41205 I0408 18:33:37.309262 6849 webserver.cc:184] Document root disabled I0408 18:33:37.309561 6849 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:41205/ I0408 18:33:37.309857 6849 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:37.310194 6849 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:37.310228 6970 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:37.310328 6970 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:37.310369 6970 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:37.313407 6864 log.cc:520] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Log is configured to *not* fsync() on all Append() calls I0408 18:33:37.314002 6969 heartbeater.cc:346] Connected to a master server at 127.6.54.62:36049 I0408 18:33:37.314049 6969 heartbeater.cc:426] Registering TS with master... I0408 18:33:37.314139 6969 heartbeater.cc:475] Master 127.6.54.62:36049 requested a full tablet report, sending... I0408 18:33:37.314574 6652 ts_manager.cc:87] Re-registered known tserver with Master: a8cbf6f597154071980a76fd950b98ca (127.6.54.1:43433) I0408 18:33:37.315065 6652 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:57383 I0408 18:33:37.318505 6864 tablet_bootstrap.cc:436] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=23 ignored=0} inserts{seen=1100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0408 18:33:37.318732 6864 tablet_bootstrap.cc:436] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Bootstrap complete. I0408 18:33:37.319300 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 6849 I0408 18:33:37.319447 6864 ts_tablet_manager.cc:955] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s I0408 18:33:37.320588 6864 raft_consensus.cc:304] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 1 FOLLOWER]: Replica starting. Triggering 2 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } } I0408 18:33:37.320937 6864 raft_consensus.cc:605] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cbf6f597154071980a76fd950b98ca, State: Initialized, Role: FOLLOWER I0408 18:33:37.321377 6864 consensus_queue.cc:226] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } } I0408 18:33:37.321410 6864 raft_consensus.cc:344] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:37.321434 6864 raft_consensus.cc:436] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:37.321487 6864 raft_consensus.cc:2700] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Snoozing failure detection for 1.727s (starting election) I0408 18:33:37.321512 6864 raft_consensus.cc:2752] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:37.322311 6864 raft_consensus.cc:458] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } } I0408 18:33:37.322361 6864 leader_election.cc:258] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:37.322489 6980 raft_consensus.cc:2700] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Snoozing failure detection for 1.886s (election complete) I0408 18:33:37.322512 6980 raft_consensus.cc:2507] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:37.322595 6980 raft_consensus.cc:572] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [term 2 LEADER]: Becoming Leader. State: Replica: a8cbf6f597154071980a76fd950b98ca, State: Running, Role: LEADER I0408 18:33:37.322625 6969 heartbeater.cc:467] Master 127.6.54.62:36049 was elected leader, sending a full tablet report... I0408 18:33:37.322667 6980 consensus_queue.cc:203] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } } I0408 18:33:37.322597 6864 ts_tablet_manager.cc:983] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s I0408 18:33:37.323141 6652 catalog_manager.cc:3595] T 2248fdd624374521b5944813156ec928 P a8cbf6f597154071980a76fd950b98ca reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a8cbf6f597154071980a76fd950b98ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8cbf6f597154071980a76fd950b98ca" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43433 } health_report { overall_health: HEALTHY } } } I0408 18:33:37.329288 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 6849 W0408 18:33:37.332998 6636 connection.cc:513] server connection from 127.6.54.1:57383 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:37.333094 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 6627 W0408 18:33:37.335621 6824 connection.cc:513] client connection to 127.6.54.62:36049 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:37.335649 6626 connection.cc:513] client connection to 127.6.54.62:36049 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestRestartWithOrphanedReplicates/0 (1422 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestRestartWithOrphanedReplicates/1 I0408 18:33:37.337941 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:37.370180 6994 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:37.370316 6994 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:37.371084 6994 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:37.371101 6994 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:37.371105 6994 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:37.372164 6994 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.6994 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:37.373037 6994 master_main.cc:73] Initializing master server... I0408 18:33:37.373237 6994 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 27511us I0408 18:33:37.373415 6994 fs_manager.cc:260] Metadata directory not provided I0408 18:33:37.373450 6994 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:37.373522 6994 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:37.373551 6994 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:37.374806 6994 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "415e3c9c65bd4884bda93b6d696ee920" format_stamp: "Formatted at 2019-04-08 18:33:37 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:37.374982 6994 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "415e3c9c65bd4884bda93b6d696ee920" format_stamp: "Formatted at 2019-04-08 18:33:37 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:37.376221 6994 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0408 18:33:37.376463 6994 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:37.376495 6994 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:37.377100 6994 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:37.377136 6994 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/wal uuid: "415e3c9c65bd4884bda93b6d696ee920" format_stamp: "Formatted at 2019-04-08 18:33:37 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:37.377178 6994 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:37.402580 6994 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:37.402678 6994 master_main.cc:76] Starting Master server... I0408 18:33:37.403213 7009 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:37.403232 7009 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:37.403237 7009 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:37.404479 6994 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:33005 I0408 18:33:37.404572 6994 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:37.404593 6994 webserver.cc:184] Document root disabled I0408 18:33:37.404789 6994 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:32797/ I0408 18:33:37.404986 6994 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:37.405740 7053 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:37.407804 7053 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920: Bootstrap starting. I0408 18:33:37.408223 7053 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920: No blocks or log segments found. Creating new log. I0408 18:33:37.408448 7053 log.cc:520] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920: Log is configured to *not* fsync() on all Append() calls I0408 18:33:37.408885 7053 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920: No bootstrap required, opened a new log I0408 18:33:37.410120 7053 raft_consensus.cc:304] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3c9c65bd4884bda93b6d696ee920" member_type: VOTER } I0408 18:33:37.410156 7053 raft_consensus.cc:330] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:37.410172 7053 raft_consensus.cc:605] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415e3c9c65bd4884bda93b6d696ee920, State: Initialized, Role: FOLLOWER I0408 18:33:37.410259 7053 consensus_queue.cc:226] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3c9c65bd4884bda93b6d696ee920" member_type: VOTER } I0408 18:33:37.410281 7053 raft_consensus.cc:344] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:37.410296 7053 raft_consensus.cc:436] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:37.410320 7053 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920: Snoozing failure detection for 1.575s (starting election) I0408 18:33:37.410331 7053 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:37.410670 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 6994 I0408 18:33:37.410902 7053 raft_consensus.cc:458] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3c9c65bd4884bda93b6d696ee920" member_type: VOTER } I0408 18:33:37.410938 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:33005 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} I0408 18:33:37.410969 7053 leader_election.cc:258] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:37.411186 7053 sys_catalog.cc:335] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3c9c65bd4884bda93b6d696ee920" member_type: VOTER } } I0408 18:33:37.411211 7053 sys_catalog.cc:338] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [sys.catalog]: This master's current role is: FOLLOWER I0408 18:33:37.411309 7053 sys_catalog.cc:422] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:37.411571 6994 master_main.cc:79] Master server successfully started. W0408 18:33:37.411772 7057 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:37.411794 7057 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:37.412245 7055 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920: Snoozing failure detection for 2.028s (election complete) I0408 18:33:37.412271 7055 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:37.412338 7055 raft_consensus.cc:572] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [term 1 LEADER]: Becoming Leader. State: Replica: 415e3c9c65bd4884bda93b6d696ee920, State: Running, Role: LEADER I0408 18:33:37.412402 7055 consensus_queue.cc:203] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3c9c65bd4884bda93b6d696ee920" member_type: VOTER } I0408 18:33:37.412616 7055 sys_catalog.cc:335] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "415e3c9c65bd4884bda93b6d696ee920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3c9c65bd4884bda93b6d696ee920" member_type: VOTER } } I0408 18:33:37.412639 7055 sys_catalog.cc:338] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [sys.catalog]: This master's current role is: LEADER I0408 18:33:37.412714 7055 sys_catalog.cc:335] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415e3c9c65bd4884bda93b6d696ee920. Latest consensus state: current_term: 1 leader_uuid: "415e3c9c65bd4884bda93b6d696ee920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3c9c65bd4884bda93b6d696ee920" member_type: VOTER } } I0408 18:33:37.412734 7055 sys_catalog.cc:338] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920 [sys.catalog]: This master's current role is: LEADER I0408 18:33:37.413974 7062 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:37.414108 7062 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0408 18:33:37.436240 7062 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:37.437208 7062 catalog_manager.cc:992] Loading token signing keys... WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:37.444790 7059 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:37.445022 7059 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:37.445744 7059 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:37.445775 7059 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:37.446709 7059 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:33005 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7059 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:37.447829 7059 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:37.448067 7059 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 27511us I0408 18:33:37.448212 7059 fs_manager.cc:260] Metadata directory not provided I0408 18:33:37.448246 7059 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:37.448323 7059 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:37.448340 7059 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:37.449601 7059 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "ec730ea11e2f448dbd771aa556aea5b7" format_stamp: "Formatted at 2019-04-08 18:33:37 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:37.449780 7059 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "ec730ea11e2f448dbd771aa556aea5b7" format_stamp: "Formatted at 2019-04-08 18:33:37 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:37.454955 7062 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 415e3c9c65bd4884bda93b6d696ee920: Generated new TSK 0 I0408 18:33:37.463431 7059 fs_manager.cc:495] Time spent creating directory manager: real 0.013s user 0.002s sys 0.000s I0408 18:33:37.463744 7059 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:37.463780 7059 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:37.464373 7059 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:37.464413 7059 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "ec730ea11e2f448dbd771aa556aea5b7" format_stamp: "Formatted at 2019-04-08 18:33:37 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:37.464467 7059 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:37.497133 7059 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:37.497339 7059 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:37.497412 7059 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:37.501775 7059 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:44705 I0408 18:33:37.501852 7059 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:37.501860 7059 webserver.cc:184] Document root disabled I0408 18:33:37.502130 7059 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36397/ I0408 18:33:37.502388 7059 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:37.502923 7059 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:37.502784 7184 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:37.503036 7184 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:37.503052 7184 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:37.505452 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 7059 I0408 18:33:37.512029 7183 heartbeater.cc:346] Connected to a master server at 127.6.54.62:33005 I0408 18:33:37.512076 7183 heartbeater.cc:426] Registering TS with master... I0408 18:33:37.512169 7183 heartbeater.cc:475] Master 127.6.54.62:33005 requested a full tablet report, sending... I0408 18:33:37.512688 7020 ts_manager.cc:81] Registered new tserver with Master: ec730ea11e2f448dbd771aa556aea5b7 (127.6.54.1:44705) I0408 18:33:37.513463 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:37.513947 7020 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:60879 I0408 18:33:37.518254 6360 test_util.cc:200] Using random seed: 321137325 I0408 18:33:37.523679 7020 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36828: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:37.530344 7117 tablet_service.cc:758] Processing CreateTablet for tablet bb51e69139f14cb2a6834cff70d9e745 (table=test-workload [id=5a6e4e61cad947e584aeadbc967d0844]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:37.530477 7117 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet bb51e69139f14cb2a6834cff70d9e745. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:37.532174 7117 ts_tablet_manager.cc:1080] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:37.532315 7201 ts_tablet_manager.cc:938] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Bootstrapping tablet I0408 18:33:37.532385 7201 tablet_bootstrap.cc:436] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Bootstrap starting. I0408 18:33:37.532781 7201 tablet_bootstrap.cc:581] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: No blocks or log segments found. Creating new log. I0408 18:33:37.533017 7201 log.cc:520] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Log is configured to *not* fsync() on all Append() calls I0408 18:33:37.533428 7201 tablet_bootstrap.cc:436] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: No bootstrap required, opened a new log I0408 18:33:37.533484 7201 ts_tablet_manager.cc:955] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:37.534488 7201 raft_consensus.cc:304] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } } I0408 18:33:37.534538 7201 raft_consensus.cc:330] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:37.534559 7201 raft_consensus.cc:605] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec730ea11e2f448dbd771aa556aea5b7, State: Initialized, Role: FOLLOWER I0408 18:33:37.534667 7201 consensus_queue.cc:226] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } } I0408 18:33:37.534690 7201 raft_consensus.cc:344] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:37.534705 7201 raft_consensus.cc:436] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:37.534730 7201 raft_consensus.cc:2700] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Snoozing failure detection for 1.511s (starting election) I0408 18:33:37.534747 7201 raft_consensus.cc:2752] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:37.535179 7201 raft_consensus.cc:458] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } } I0408 18:33:37.535243 7201 leader_election.cc:258] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:37.535383 7203 raft_consensus.cc:2700] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Snoozing failure detection for 1.758s (election complete) I0408 18:33:37.535409 7203 raft_consensus.cc:2507] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:37.535459 7183 heartbeater.cc:467] Master 127.6.54.62:33005 was elected leader, sending a full tablet report... I0408 18:33:37.535473 7201 ts_tablet_manager.cc:983] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:37.535476 7203 raft_consensus.cc:572] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 1 LEADER]: Becoming Leader. State: Replica: ec730ea11e2f448dbd771aa556aea5b7, State: Running, Role: LEADER I0408 18:33:37.535611 7203 consensus_queue.cc:203] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } } I0408 18:33:37.536242 7020 catalog_manager.cc:3595] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 reported cstate change: term changed from 0 to 1, leader changed from to ec730ea11e2f448dbd771aa556aea5b7 (127.6.54.1). New cstate: current_term: 1 leader_uuid: "ec730ea11e2f448dbd771aa556aea5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } health_report { overall_health: HEALTHY } } } I0408 18:33:37.539963 7180 generic_service.cc:146] {username='slave'} at 127.0.0.1:53106 changed flags via RPC: fault_crash_before_append_commit from '0' to '0.05' W0408 18:33:37.551740 7214 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:37.551793 7214 fault_injection.cc:42] THIS SERVER MAY CRASH! E0408 18:33:37.554586 7213 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_append_commit (process will exit) W0408 18:33:37.557762 7197 connection.cc:513] client connection to 127.6.54.1:44705 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:37.557786 6993 connection.cc:513] client connection to 127.6.54.1:44705 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:37.557796 7197 meta_cache.cc:219] Tablet bb51e69139f14cb2a6834cff70d9e745: Replica ec730ea11e2f448dbd771aa556aea5b7 (127.6.54.1:44705) has failed: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:37.557832 7004 connection.cc:513] server connection from 127.6.54.1:60879 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:37.557874 7197 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0408 18:33:37.557940 7005 connection.cc:513] client connection to 127.6.54.1:44705 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:38.555675 7197 batcher.cc:349] Timed out: Failed to write batch of 50 ops to tablet bb51e69139f14cb2a6834cff70d9e745 after 41 attempt(s): Failed to write to server: (no server available): Write(tablet: bb51e69139f14cb2a6834cff70d9e745, num_ops: 50, num_attempts: 41) passed its deadline: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:38.589705 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:44705 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=36397 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:33005 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:38.622391 7217 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:38.622546 7217 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:38.623236 7217 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:38.623260 7217 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:38.624209 7217 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:44705 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=36397 --tserver_master_addrs=127.6.54.62:33005 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7217 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:38.625182 7217 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:38.625421 7217 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28011us I0408 18:33:38.625576 7217 fs_manager.cc:260] Metadata directory not provided I0408 18:33:38.625627 7217 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:38.626610 7217 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:38.626821 7217 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:38.626852 7217 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:38.627470 7217 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:38.627852 7217 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "ec730ea11e2f448dbd771aa556aea5b7" format_stamp: "Formatted at 2019-04-08 18:33:37 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:38.627902 7217 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:38.659611 7217 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:38.659837 7217 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:38.659860 7217 ts_tablet_manager.cc:915] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Loading tablet metadata I0408 18:33:38.660459 7217 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:38.660979 7217 ts_tablet_manager.cc:1080] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:38.661135 7232 ts_tablet_manager.cc:938] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Bootstrapping tablet I0408 18:33:38.661147 7217 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:38.661237 7232 tablet_bootstrap.cc:436] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Bootstrap starting. I0408 18:33:38.666826 7217 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:44705 I0408 18:33:38.666914 7217 webserver.cc:173] Starting webserver on 127.6.54.1:36397 I0408 18:33:38.666929 7217 webserver.cc:184] Document root disabled I0408 18:33:38.667136 7217 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36397/ I0408 18:33:38.667476 7217 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithOrphanedReplicates_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:38.667773 7217 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:38.667930 7338 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:38.667958 7338 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:38.667965 7338 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:38.669518 7232 log.cc:520] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Log is configured to *not* fsync() on all Append() calls I0408 18:33:38.672086 7337 heartbeater.cc:346] Connected to a master server at 127.6.54.62:33005 I0408 18:33:38.672135 7337 heartbeater.cc:426] Registering TS with master... I0408 18:33:38.672233 7337 heartbeater.cc:475] Master 127.6.54.62:33005 requested a full tablet report, sending... I0408 18:33:38.672637 7232 tablet_bootstrap.cc:436] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=10 ignored=0} inserts{seen=450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0408 18:33:38.672686 7019 ts_manager.cc:87] Re-registered known tserver with Master: ec730ea11e2f448dbd771aa556aea5b7 (127.6.54.1:44705) I0408 18:33:38.672929 7232 tablet_bootstrap.cc:436] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Bootstrap complete. I0408 18:33:38.672922 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 7217 I0408 18:33:38.673450 7019 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:42337 I0408 18:33:38.673949 7232 ts_tablet_manager.cc:955] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s I0408 18:33:38.674984 7232 raft_consensus.cc:304] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } } I0408 18:33:38.675549 7232 raft_consensus.cc:605] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec730ea11e2f448dbd771aa556aea5b7, State: Initialized, Role: FOLLOWER I0408 18:33:38.675698 7232 consensus_queue.cc:226] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } } I0408 18:33:38.675727 7232 raft_consensus.cc:344] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:38.675750 7232 raft_consensus.cc:436] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:38.675770 7232 raft_consensus.cc:2700] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Snoozing failure detection for 1.926s (starting election) I0408 18:33:38.675778 7232 raft_consensus.cc:2752] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:38.676538 7232 raft_consensus.cc:458] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } } I0408 18:33:38.676590 7232 leader_election.cc:258] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:38.676734 7348 raft_consensus.cc:2700] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Snoozing failure detection for 2.188s (election complete) I0408 18:33:38.676760 7348 raft_consensus.cc:2507] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:38.676822 7232 ts_tablet_manager.cc:983] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I0408 18:33:38.676829 7348 raft_consensus.cc:572] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [term 2 LEADER]: Becoming Leader. State: Replica: ec730ea11e2f448dbd771aa556aea5b7, State: Running, Role: LEADER I0408 18:33:38.677016 7348 consensus_queue.cc:203] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } } I0408 18:33:38.676895 7337 heartbeater.cc:467] Master 127.6.54.62:33005 was elected leader, sending a full tablet report... I0408 18:33:38.677500 7019 catalog_manager.cc:3595] T bb51e69139f14cb2a6834cff70d9e745 P ec730ea11e2f448dbd771aa556aea5b7 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "ec730ea11e2f448dbd771aa556aea5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec730ea11e2f448dbd771aa556aea5b7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44705 } health_report { overall_health: HEALTHY } } } I0408 18:33:38.682714 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 7217 W0408 18:33:38.686843 7003 connection.cc:513] server connection from 127.6.54.1:42337 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:38.686964 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 6994 W0408 18:33:38.689580 7190 connection.cc:513] client connection to 127.6.54.62:33005 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:38.689651 6993 connection.cc:513] client connection to 127.6.54.62:33005 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestRestartWithOrphanedReplicates/1 (1354 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp/0 I0408 18:33:38.691941 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data --block_manager=file --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:38.724401 7362 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:38.724560 7362 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:38.725302 7362 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:38.725318 7362 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:38.725322 7362 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:38.726378 7362 master_main.cc:66] Master server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.7362 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:38.727303 7362 master_main.cc:73] Initializing master server... I0408 18:33:38.727506 7362 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28011us I0408 18:33:38.727689 7362 fs_manager.cc:260] Metadata directory not provided I0408 18:33:38.727708 7362 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:38.727766 7362 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:38.727774 7362 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:38.729030 7362 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "4eb040d232ce43d6a59499f86ee8f7a9" format_stamp: "Formatted at 2019-04-08 18:33:38 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:38.729189 7362 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "4eb040d232ce43d6a59499f86ee8f7a9" format_stamp: "Formatted at 2019-04-08 18:33:38 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:38.730294 7362 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:38.730502 7362 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:38.730540 7362 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:38.730621 7362 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:38.730656 7362 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/wal uuid: "4eb040d232ce43d6a59499f86ee8f7a9" format_stamp: "Formatted at 2019-04-08 18:33:38 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:38.730686 7362 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:38.756904 7362 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:38.756975 7362 master_main.cc:76] Starting Master server... I0408 18:33:38.757138 7377 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:38.757164 7377 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:38.757169 7377 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:38.759366 7362 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:45097 I0408 18:33:38.759426 7362 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:38.759460 7362 webserver.cc:184] Document root disabled I0408 18:33:38.759677 7362 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:40677/ I0408 18:33:38.759907 7362 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:38.760341 7421 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:38.762452 7421 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9: Bootstrap starting. I0408 18:33:38.762908 7421 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9: No blocks or log segments found. Creating new log. I0408 18:33:38.763168 7421 log.cc:520] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9: Log is configured to *not* fsync() on all Append() calls I0408 18:33:38.763566 7421 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9: No bootstrap required, opened a new log I0408 18:33:38.764608 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 7362 I0408 18:33:38.764734 7421 raft_consensus.cc:304] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" member_type: VOTER } I0408 18:33:38.764771 7421 raft_consensus.cc:330] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:38.764784 7421 raft_consensus.cc:605] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb040d232ce43d6a59499f86ee8f7a9, State: Initialized, Role: FOLLOWER I0408 18:33:38.764869 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:45097 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} I0408 18:33:38.764895 7421 consensus_queue.cc:226] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" member_type: VOTER } I0408 18:33:38.764914 7421 raft_consensus.cc:344] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:38.764927 7421 raft_consensus.cc:436] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:38.764953 7421 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9: Snoozing failure detection for 2.134s (starting election) I0408 18:33:38.764963 7421 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:38.765367 7421 raft_consensus.cc:458] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" member_type: VOTER } I0408 18:33:38.765429 7421 leader_election.cc:258] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:38.765589 7423 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9: Snoozing failure detection for 1.822s (election complete) I0408 18:33:38.765615 7423 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:38.765691 7423 raft_consensus.cc:572] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb040d232ce43d6a59499f86ee8f7a9, State: Running, Role: LEADER I0408 18:33:38.766448 7423 consensus_queue.cc:203] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" member_type: VOTER } I0408 18:33:38.766789 7425 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eb040d232ce43d6a59499f86ee8f7a9. Latest consensus state: current_term: 1 leader_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" member_type: VOTER } } I0408 18:33:38.766878 7425 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [sys.catalog]: This master's current role is: LEADER I0408 18:33:38.767032 7421 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" member_type: VOTER } } I0408 18:33:38.767112 7421 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [sys.catalog]: This master's current role is: LEADER I0408 18:33:38.767124 7424 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb040d232ce43d6a59499f86ee8f7a9" member_type: VOTER } } I0408 18:33:38.767179 7424 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [sys.catalog]: This master's current role is: LEADER I0408 18:33:38.767285 7421 sys_catalog.cc:422] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:38.767505 7362 master_main.cc:79] Master server successfully started. W0408 18:33:38.767727 7429 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:38.767752 7429 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:38.767803 7428 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:38.767930 7428 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0408 18:33:38.785434 7428 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:38.786033 7428 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:38.789832 7428 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 4eb040d232ce43d6a59499f86ee8f7a9: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:38.812553 7426 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:38.812757 7426 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:38.813771 7426 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:38.813807 7426 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:38.815243 7426 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:45097 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7426 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:38.816680 7426 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:38.816998 7426 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28011us I0408 18:33:38.817193 7426 fs_manager.cc:260] Metadata directory not provided I0408 18:33:38.817242 7426 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:38.817384 7426 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:38.817415 7426 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:38.819566 7426 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "cbd3ec9db28e48b78d32958ae4f09054" format_stamp: "Formatted at 2019-04-08 18:33:38 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:38.821300 7426 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "cbd3ec9db28e48b78d32958ae4f09054" format_stamp: "Formatted at 2019-04-08 18:33:38 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:38.822948 7426 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:38.823283 7426 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:38.823319 7426 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:38.823427 7426 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:38.823467 7426 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "cbd3ec9db28e48b78d32958ae4f09054" format_stamp: "Formatted at 2019-04-08 18:33:38 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:38.823513 7426 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:38.896814 7426 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:38.897079 7426 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:38.897156 7426 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:38.901614 7426 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:43563 I0408 18:33:38.901702 7426 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:38.901720 7426 webserver.cc:184] Document root disabled I0408 18:33:38.901967 7426 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36765/ I0408 18:33:38.902220 7426 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:38.902613 7426 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:38.902627 7552 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:38.902639 7552 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:38.902645 7552 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:38.907292 7551 heartbeater.cc:346] Connected to a master server at 127.6.54.62:45097 I0408 18:33:38.907333 7551 heartbeater.cc:426] Registering TS with master... I0408 18:33:38.907429 7551 heartbeater.cc:475] Master 127.6.54.62:45097 requested a full tablet report, sending... I0408 18:33:38.907907 7388 ts_manager.cc:81] Registered new tserver with Master: cbd3ec9db28e48b78d32958ae4f09054 (127.6.54.1:43563) I0408 18:33:38.908677 7388 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:40345 I0408 18:33:38.908901 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 7426 I0408 18:33:38.911554 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:38.918043 7548 generic_service.cc:146] {username='slave'} at 127.0.0.1:36112 changed flags via RPC: fault_crash_before_append_commit from '0' to '0.01' I0408 18:33:38.918298 6360 test_util.cc:200] Using random seed: 322537369 I0408 18:33:38.922868 7388 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47108: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:38.927207 7488 tablet_service.cc:758] Processing CreateTablet for tablet eed5de6e1d21458e89aaad2d796a4b97 (table=test-workload [id=19decff4bd8a40168aa1135e569fee2b]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:38.927378 7488 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet eed5de6e1d21458e89aaad2d796a4b97. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:38.929352 7488 ts_tablet_manager.cc:1080] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:38.929538 7568 ts_tablet_manager.cc:938] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Bootstrapping tablet I0408 18:33:38.929615 7568 tablet_bootstrap.cc:436] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Bootstrap starting. I0408 18:33:38.929952 7568 tablet_bootstrap.cc:581] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: No blocks or log segments found. Creating new log. I0408 18:33:38.930209 7568 log.cc:520] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Log is configured to *not* fsync() on all Append() calls I0408 18:33:38.930666 7568 tablet_bootstrap.cc:436] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: No bootstrap required, opened a new log I0408 18:33:38.930721 7568 ts_tablet_manager.cc:955] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:38.931948 7568 raft_consensus.cc:304] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } } I0408 18:33:38.931985 7568 raft_consensus.cc:330] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:38.931998 7568 raft_consensus.cc:605] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd3ec9db28e48b78d32958ae4f09054, State: Initialized, Role: FOLLOWER I0408 18:33:38.932102 7568 consensus_queue.cc:226] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } } I0408 18:33:38.932123 7568 raft_consensus.cc:344] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:38.932133 7568 raft_consensus.cc:436] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:38.932157 7568 raft_consensus.cc:2700] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Snoozing failure detection for 2.167s (starting election) I0408 18:33:38.932174 7568 raft_consensus.cc:2752] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:38.932720 7568 raft_consensus.cc:458] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } } I0408 18:33:38.932773 7568 leader_election.cc:258] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:38.932943 7570 raft_consensus.cc:2700] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Snoozing failure detection for 1.650s (election complete) I0408 18:33:38.932968 7570 raft_consensus.cc:2507] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:38.933006 7551 heartbeater.cc:467] Master 127.6.54.62:45097 was elected leader, sending a full tablet report... I0408 18:33:38.933039 7570 raft_consensus.cc:572] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 1 LEADER]: Becoming Leader. State: Replica: cbd3ec9db28e48b78d32958ae4f09054, State: Running, Role: LEADER I0408 18:33:38.933162 7568 ts_tablet_manager.cc:983] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s I0408 18:33:38.933217 7570 consensus_queue.cc:203] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } } W0408 18:33:38.933776 7570 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:38.933794 7570 fault_injection.cc:42] THIS SERVER MAY CRASH! I0408 18:33:38.934034 7388 catalog_manager.cc:3595] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 reported cstate change: term changed from 0 to 1, leader changed from to cbd3ec9db28e48b78d32958ae4f09054 (127.6.54.1). New cstate: current_term: 1 leader_uuid: "cbd3ec9db28e48b78d32958ae4f09054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } health_report { overall_health: HEALTHY } } } E0408 18:33:38.957011 7599 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_append_commit (process will exit) W0408 18:33:38.960683 7564 connection.cc:513] client connection to 127.6.54.1:43563 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:38.960723 7564 meta_cache.cc:219] Tablet eed5de6e1d21458e89aaad2d796a4b97: Replica cbd3ec9db28e48b78d32958ae4f09054 (127.6.54.1:43563) has failed: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:38.960743 7361 connection.cc:513] client connection to 127.6.54.1:43563 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:38.960737 7372 connection.cc:513] client connection to 127.6.54.1:43563 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:38.960844 7564 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0408 18:33:38.960947 7372 connection.cc:513] server connection from 127.6.54.1:40345 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.056835 7562 meta_cache.cc:818] Timed out: GetTableLocations { table: 'test-workload', partition-key: (), attempt: 1 } failed: timed out after deadline expired: GetTableLocations RPC to 127.6.54.62:45097 timed out after 0.000s (ON_OUTBOUND_QUEUE) I0408 18:33:39.070925 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:43563 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=36765 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:45097 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:39.104058 7600 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:39.104195 7600 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:39.104854 7600 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:39.104876 7600 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:39.105800 7600 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:43563 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=36765 --tserver_master_addrs=127.6.54.62:45097 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7600 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:39.106732 7600 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:39.106967 7600 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28511us I0408 18:33:39.107106 7600 fs_manager.cc:260] Metadata directory not provided I0408 18:33:39.107146 7600 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:39.108156 7600 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s I0408 18:33:39.108356 7600 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:39.108382 7600 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:39.108448 7600 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:39.108824 7600 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "cbd3ec9db28e48b78d32958ae4f09054" format_stamp: "Formatted at 2019-04-08 18:33:38 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.108867 7600 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:39.139600 7600 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:39.139847 7600 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:39.139866 7600 ts_tablet_manager.cc:915] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Loading tablet metadata I0408 18:33:39.140441 7600 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:39.140969 7600 ts_tablet_manager.cc:1080] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:39.141096 7600 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:39.141096 7615 ts_tablet_manager.cc:938] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Bootstrapping tablet I0408 18:33:39.141227 7615 tablet_bootstrap.cc:436] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Bootstrap starting. I0408 18:33:39.147181 7600 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:43563 I0408 18:33:39.147272 7600 webserver.cc:173] Starting webserver on 127.6.54.1:36765 I0408 18:33:39.147289 7600 webserver.cc:184] Document root disabled I0408 18:33:39.147524 7600 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36765/ I0408 18:33:39.147770 7600 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:39.148114 7600 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:39.148468 7721 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:39.148491 7721 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:39.148497 7721 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:39.150004 7615 log.cc:520] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Log is configured to *not* fsync() on all Append() calls I0408 18:33:39.151959 7720 heartbeater.cc:346] Connected to a master server at 127.6.54.62:45097 I0408 18:33:39.151998 7720 heartbeater.cc:426] Registering TS with master... I0408 18:33:39.152078 7720 heartbeater.cc:475] Master 127.6.54.62:45097 requested a full tablet report, sending... I0408 18:33:39.152518 7387 ts_manager.cc:87] Re-registered known tserver with Master: cbd3ec9db28e48b78d32958ae4f09054 (127.6.54.1:43563) I0408 18:33:39.153017 7387 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:40709 I0408 18:33:39.153990 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 7600 I0408 18:33:39.155793 7615 tablet_bootstrap.cc:436] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Bootstrap replayed 1/1 log segments. Stats: ops{read=89 overwritten=0 applied=86 ignored=65} inserts{seen=20 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates I0408 18:33:39.156054 7615 tablet_bootstrap.cc:436] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Bootstrap complete. I0408 18:33:39.156911 7615 ts_tablet_manager.cc:955] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s I0408 18:33:39.157984 7615 raft_consensus.cc:304] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 1 FOLLOWER]: Replica starting. Triggering 3 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } } I0408 18:33:39.158246 7615 raft_consensus.cc:605] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd3ec9db28e48b78d32958ae4f09054, State: Initialized, Role: FOLLOWER I0408 18:33:39.158370 7615 consensus_queue.cc:226] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 1.89, Last appended by leader: 89, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } } I0408 18:33:39.158396 7615 raft_consensus.cc:344] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:39.158417 7615 raft_consensus.cc:436] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:39.158444 7615 raft_consensus.cc:2700] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Snoozing failure detection for 1.850s (starting election) I0408 18:33:39.158454 7615 raft_consensus.cc:2752] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:39.159274 7615 raft_consensus.cc:458] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } } I0408 18:33:39.159333 7615 leader_election.cc:258] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:39.159467 7732 raft_consensus.cc:2700] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Snoozing failure detection for 1.723s (election complete) I0408 18:33:39.159490 7732 raft_consensus.cc:2507] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:39.159570 7615 ts_tablet_manager.cc:983] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s I0408 18:33:39.159641 7720 heartbeater.cc:467] Master 127.6.54.62:45097 was elected leader, sending a full tablet report... I0408 18:33:39.159580 7732 raft_consensus.cc:572] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [term 2 LEADER]: Becoming Leader. State: Replica: cbd3ec9db28e48b78d32958ae4f09054, State: Running, Role: LEADER I0408 18:33:39.159765 7732 consensus_queue.cc:203] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.89, Last appended by leader: 89, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } } I0408 18:33:39.160225 7387 catalog_manager.cc:3595] T eed5de6e1d21458e89aaad2d796a4b97 P cbd3ec9db28e48b78d32958ae4f09054 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "cbd3ec9db28e48b78d32958ae4f09054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd3ec9db28e48b78d32958ae4f09054" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43563 } health_report { overall_health: HEALTHY } } } I0408 18:33:39.162483 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 7600 W0408 18:33:39.165849 7372 connection.cc:513] server connection from 127.6.54.1:40709 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:39.165956 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 7362 W0408 18:33:39.168654 7556 connection.cc:513] client connection to 127.6.54.62:45097 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.168666 7361 connection.cc:513] client connection to 127.6.54.62:45097 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp/0 (479 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp/1 I0408 18:33:39.171056 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:39.203837 7741 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:39.203984 7741 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:39.204731 7741 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:39.204746 7741 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:39.204749 7741 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:39.205770 7741 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.7741 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:39.206634 7741 master_main.cc:73] Initializing master server... I0408 18:33:39.206827 7741 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28511us I0408 18:33:39.207010 7741 fs_manager.cc:260] Metadata directory not provided I0408 18:33:39.207038 7741 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:39.207123 7741 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:39.207141 7741 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:39.208431 7741 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "f5c4337061a244b098fbe7e9ced19cf4" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.208614 7741 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "f5c4337061a244b098fbe7e9ced19cf4" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.209820 7741 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:39.210084 7741 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:39.210114 7741 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:39.210700 7741 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:39.210732 7741 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/wal uuid: "f5c4337061a244b098fbe7e9ced19cf4" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.210772 7741 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:39.221088 7741 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:39.221151 7741 master_main.cc:76] Starting Master server... I0408 18:33:39.221253 7756 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:39.221273 7756 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:39.221278 7756 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:39.223598 7741 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:34269 I0408 18:33:39.223659 7741 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:39.223673 7741 webserver.cc:184] Document root disabled I0408 18:33:39.223862 7741 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:36449/ I0408 18:33:39.224066 7741 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:39.224550 7800 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:39.226585 7800 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4: Bootstrap starting. I0408 18:33:39.226979 7800 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4: No blocks or log segments found. Creating new log. I0408 18:33:39.227226 7800 log.cc:520] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4: Log is configured to *not* fsync() on all Append() calls I0408 18:33:39.227602 7800 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4: No bootstrap required, opened a new log I0408 18:33:39.228741 7800 raft_consensus.cc:304] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c4337061a244b098fbe7e9ced19cf4" member_type: VOTER } I0408 18:33:39.228776 7800 raft_consensus.cc:330] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:39.228790 7800 raft_consensus.cc:605] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5c4337061a244b098fbe7e9ced19cf4, State: Initialized, Role: FOLLOWER I0408 18:33:39.228891 7800 consensus_queue.cc:226] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c4337061a244b098fbe7e9ced19cf4" member_type: VOTER } I0408 18:33:39.228919 7800 raft_consensus.cc:344] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:39.228931 7800 raft_consensus.cc:436] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:39.228948 7800 raft_consensus.cc:2700] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4: Snoozing failure detection for 1.642s (starting election) I0408 18:33:39.228965 7800 raft_consensus.cc:2752] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:39.229346 7800 raft_consensus.cc:458] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c4337061a244b098fbe7e9ced19cf4" member_type: VOTER } I0408 18:33:39.229394 7800 leader_election.cc:258] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:39.229518 7802 raft_consensus.cc:2700] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4: Snoozing failure detection for 2.090s (election complete) I0408 18:33:39.229552 7802 raft_consensus.cc:2507] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:39.229640 7802 raft_consensus.cc:572] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [term 1 LEADER]: Becoming Leader. State: Replica: f5c4337061a244b098fbe7e9ced19cf4, State: Running, Role: LEADER I0408 18:33:39.229713 7802 consensus_queue.cc:203] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c4337061a244b098fbe7e9ced19cf4" member_type: VOTER } I0408 18:33:39.229912 7802 sys_catalog.cc:335] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5c4337061a244b098fbe7e9ced19cf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c4337061a244b098fbe7e9ced19cf4" member_type: VOTER } } I0408 18:33:39.229926 7800 sys_catalog.cc:335] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "f5c4337061a244b098fbe7e9ced19cf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c4337061a244b098fbe7e9ced19cf4" member_type: VOTER } } I0408 18:33:39.229935 7802 sys_catalog.cc:338] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [sys.catalog]: This master's current role is: LEADER I0408 18:33:39.229939 7800 sys_catalog.cc:338] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [sys.catalog]: This master's current role is: LEADER I0408 18:33:39.230018 7802 sys_catalog.cc:335] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5c4337061a244b098fbe7e9ced19cf4. Latest consensus state: current_term: 1 leader_uuid: "f5c4337061a244b098fbe7e9ced19cf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c4337061a244b098fbe7e9ced19cf4" member_type: VOTER } } I0408 18:33:39.230034 7802 sys_catalog.cc:338] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [sys.catalog]: This master's current role is: LEADER I0408 18:33:39.230031 7800 sys_catalog.cc:422] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:39.230319 7741 master_main.cc:79] Master server successfully started. W0408 18:33:39.230644 7807 catalog_manager.cc:1018] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:39.230672 7807 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:39.230720 7806 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:39.230847 7806 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0408 18:33:39.233656 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 7741 I0408 18:33:39.233950 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:34269 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} I0408 18:33:39.255424 7806 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:39.256021 7806 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:39.271587 7806 catalog_manager.cc:3871] T 00000000000000000000000000000000 P f5c4337061a244b098fbe7e9ced19cf4: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:39.280376 7809 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:39.280568 7809 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:39.281580 7809 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:39.281621 7809 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:39.283018 7809 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:34269 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7809 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:39.284396 7809 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:39.284718 7809 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28511us I0408 18:33:39.284895 7809 fs_manager.cc:260] Metadata directory not provided I0408 18:33:39.284935 7809 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:39.285018 7809 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:39.285043 7809 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:39.286893 7809 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "344e933915fa48d688b5b4312ecda017" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.287107 7809 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "344e933915fa48d688b5b4312ecda017" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.291445 7809 fs_manager.cc:495] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s I0408 18:33:39.291824 7809 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:39.291862 7809 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:39.292686 7809 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:39.292726 7809 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "344e933915fa48d688b5b4312ecda017" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.292783 7809 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:39.328661 7809 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:39.328919 7809 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:39.329001 7809 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:39.333703 7809 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:42719 I0408 18:33:39.333762 7809 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:39.333770 7809 webserver.cc:184] Document root disabled I0408 18:33:39.333992 7809 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:45563/ I0408 18:33:39.334213 7809 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:39.334519 7809 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:39.334730 7931 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:39.334748 7931 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:39.334753 7931 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:39.337931 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 7809 I0408 18:33:39.340667 7930 heartbeater.cc:346] Connected to a master server at 127.6.54.62:34269 I0408 18:33:39.340718 7930 heartbeater.cc:426] Registering TS with master... I0408 18:33:39.340817 7930 heartbeater.cc:475] Master 127.6.54.62:34269 requested a full tablet report, sending... I0408 18:33:39.341329 7767 ts_manager.cc:81] Registered new tserver with Master: 344e933915fa48d688b5b4312ecda017 (127.6.54.1:42719) I0408 18:33:39.342084 7767 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:46781 I0408 18:33:39.342859 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:39.353256 7925 generic_service.cc:146] {username='slave'} at 127.0.0.1:56968 changed flags via RPC: fault_crash_before_append_commit from '0' to '0.01' I0408 18:33:39.353510 6360 test_util.cc:200] Using random seed: 322972581 I0408 18:33:39.359338 7767 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49094: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:39.365713 7859 tablet_service.cc:758] Processing CreateTablet for tablet 2285a2fec8af43fab73d1b283957292a (table=test-workload [id=f61a858dd3384e768b5d4fb44ae5f3a2]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:39.365885 7859 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 2285a2fec8af43fab73d1b283957292a. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:39.368202 7859 ts_tablet_manager.cc:1080] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:39.368353 7948 ts_tablet_manager.cc:938] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Bootstrapping tablet I0408 18:33:39.368433 7948 tablet_bootstrap.cc:436] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Bootstrap starting. I0408 18:33:39.368831 7948 tablet_bootstrap.cc:581] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: No blocks or log segments found. Creating new log. I0408 18:33:39.369058 7948 log.cc:520] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Log is configured to *not* fsync() on all Append() calls I0408 18:33:39.369452 7948 tablet_bootstrap.cc:436] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: No bootstrap required, opened a new log I0408 18:33:39.369505 7948 ts_tablet_manager.cc:955] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:39.370508 7948 raft_consensus.cc:304] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } } I0408 18:33:39.370555 7948 raft_consensus.cc:330] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:39.370569 7948 raft_consensus.cc:605] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344e933915fa48d688b5b4312ecda017, State: Initialized, Role: FOLLOWER I0408 18:33:39.370671 7948 consensus_queue.cc:226] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } } I0408 18:33:39.370692 7948 raft_consensus.cc:344] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:39.370702 7948 raft_consensus.cc:436] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:39.370721 7948 raft_consensus.cc:2700] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Snoozing failure detection for 1.937s (starting election) I0408 18:33:39.370738 7948 raft_consensus.cc:2752] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:39.371146 7948 raft_consensus.cc:458] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } } I0408 18:33:39.371234 7948 leader_election.cc:258] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:39.371395 7950 raft_consensus.cc:2700] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Snoozing failure detection for 2.052s (election complete) I0408 18:33:39.371420 7950 raft_consensus.cc:2507] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:39.371475 7930 heartbeater.cc:467] Master 127.6.54.62:34269 was elected leader, sending a full tablet report... I0408 18:33:39.371487 7948 ts_tablet_manager.cc:983] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:39.371487 7950 raft_consensus.cc:572] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 1 LEADER]: Becoming Leader. State: Replica: 344e933915fa48d688b5b4312ecda017, State: Running, Role: LEADER I0408 18:33:39.371649 7950 consensus_queue.cc:203] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } } W0408 18:33:39.372159 7950 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:39.372176 7950 fault_injection.cc:42] THIS SERVER MAY CRASH! I0408 18:33:39.372299 7767 catalog_manager.cc:3595] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 reported cstate change: term changed from 0 to 1, leader changed from to 344e933915fa48d688b5b4312ecda017 (127.6.54.1). New cstate: current_term: 1 leader_uuid: "344e933915fa48d688b5b4312ecda017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } health_report { overall_health: HEALTHY } } } E0408 18:33:39.389801 7976 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_append_commit (process will exit) W0408 18:33:39.393117 7942 connection.cc:513] client connection to 127.6.54.1:42719 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.393149 7740 connection.cc:513] client connection to 127.6.54.1:42719 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.393141 7749 connection.cc:513] client connection to 127.6.54.1:42719 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.393255 7942 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0408 18:33:39.393330 7749 connection.cc:513] server connection from 127.6.54.1:46781 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:39.506420 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:42719 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=45563 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:34269 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:39.538700 7980 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:39.538838 7980 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:39.539546 7980 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:39.539572 7980 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:39.540534 7980 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:42719 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=45563 --tserver_master_addrs=127.6.54.62:34269 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7980 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:39.541496 7980 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:39.541746 7980 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28511us I0408 18:33:39.541887 7980 fs_manager.cc:260] Metadata directory not provided I0408 18:33:39.541932 7980 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:39.542902 7980 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:39.543125 7980 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:39.543179 7980 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:39.543880 7980 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:39.544239 7980 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "344e933915fa48d688b5b4312ecda017" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.544289 7980 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:39.576023 7980 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:39.576258 7980 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:39.576282 7980 ts_tablet_manager.cc:915] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Loading tablet metadata I0408 18:33:39.576900 7980 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:39.577409 7980 ts_tablet_manager.cc:1080] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:39.577570 7980 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:39.577574 7995 ts_tablet_manager.cc:938] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Bootstrapping tablet I0408 18:33:39.577699 7995 tablet_bootstrap.cc:436] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Bootstrap starting. I0408 18:33:39.582777 7980 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:42719 I0408 18:33:39.582841 7980 webserver.cc:173] Starting webserver on 127.6.54.1:45563 I0408 18:33:39.582847 7980 webserver.cc:184] Document root disabled I0408 18:33:39.583060 7980 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:45563/ I0408 18:33:39.583420 7980 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:39.583806 7980 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:39.583829 8101 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:39.583899 8101 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:39.583906 8101 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:39.586297 7995 log.cc:520] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Log is configured to *not* fsync() on all Append() calls I0408 18:33:39.588618 8100 heartbeater.cc:346] Connected to a master server at 127.6.54.62:34269 I0408 18:33:39.588659 8100 heartbeater.cc:426] Registering TS with master... I0408 18:33:39.588742 8100 heartbeater.cc:475] Master 127.6.54.62:34269 requested a full tablet report, sending... I0408 18:33:39.589180 7767 ts_manager.cc:87] Re-registered known tserver with Master: 344e933915fa48d688b5b4312ecda017 (127.6.54.1:42719) I0408 18:33:39.589701 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 7980 I0408 18:33:39.589743 7767 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:33249 I0408 18:33:39.591712 7995 tablet_bootstrap.cc:436] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=62 ignored=42} inserts{seen=19 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0408 18:33:39.591970 7995 tablet_bootstrap.cc:436] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Bootstrap complete. I0408 18:33:39.592937 7995 ts_tablet_manager.cc:955] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s I0408 18:33:39.594449 7995 raft_consensus.cc:304] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } } I0408 18:33:39.594774 7995 raft_consensus.cc:605] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344e933915fa48d688b5b4312ecda017, State: Initialized, Role: FOLLOWER I0408 18:33:39.594902 7995 consensus_queue.cc:226] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } } I0408 18:33:39.594929 7995 raft_consensus.cc:344] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:39.594964 7995 raft_consensus.cc:436] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:39.594993 7995 raft_consensus.cc:2700] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Snoozing failure detection for 1.547s (starting election) I0408 18:33:39.595016 7995 raft_consensus.cc:2752] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:39.596325 7995 raft_consensus.cc:458] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } } I0408 18:33:39.596385 7995 leader_election.cc:258] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:39.596585 8100 heartbeater.cc:467] Master 127.6.54.62:34269 was elected leader, sending a full tablet report... I0408 18:33:39.596606 7995 ts_tablet_manager.cc:983] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s I0408 18:33:39.596683 8115 raft_consensus.cc:2700] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017: Snoozing failure detection for 1.773s (election complete) I0408 18:33:39.596709 8115 raft_consensus.cc:2507] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:39.596784 8115 raft_consensus.cc:572] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [term 2 LEADER]: Becoming Leader. State: Replica: 344e933915fa48d688b5b4312ecda017, State: Running, Role: LEADER I0408 18:33:39.596854 8115 consensus_queue.cc:203] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } } I0408 18:33:39.597424 7767 catalog_manager.cc:3595] T 2285a2fec8af43fab73d1b283957292a P 344e933915fa48d688b5b4312ecda017 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "344e933915fa48d688b5b4312ecda017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e933915fa48d688b5b4312ecda017" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42719 } health_report { overall_health: HEALTHY } } } I0408 18:33:39.598767 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 7980 W0408 18:33:39.602345 7750 connection.cc:513] server connection from 127.6.54.1:33249 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:39.602404 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 7741 W0408 18:33:39.605037 7937 connection.cc:513] client connection to 127.6.54.62:34269 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.605058 7740 connection.cc:513] client connection to 127.6.54.62:34269 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestRestartWithPendingCommitFromFailedOp/1 (437 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestCrashDuringLogReplay/0 I0408 18:33:39.607734 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data --block_manager=file --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:39.640485 8120 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:39.640645 8120 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:39.641366 8120 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:39.641383 8120 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:39.641388 8120 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:39.642410 8120 master_main.cc:66] Master server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.8120 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:39.643326 8120 master_main.cc:73] Initializing master server... I0408 18:33:39.643538 8120 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28511us I0408 18:33:39.643699 8120 fs_manager.cc:260] Metadata directory not provided I0408 18:33:39.643728 8120 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:39.643781 8120 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:39.643797 8120 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:39.645033 8120 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "6499991a1c95436295562bcb3be311e3" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.645191 8120 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "6499991a1c95436295562bcb3be311e3" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.646282 8120 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:39.646481 8120 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:39.646507 8120 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:39.646594 8120 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:39.646628 8120 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/wal uuid: "6499991a1c95436295562bcb3be311e3" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.646667 8120 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:39.681303 8120 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:39.681375 8120 master_main.cc:76] Starting Master server... I0408 18:33:39.682451 8135 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:39.682476 8135 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:39.682482 8135 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:39.683584 8120 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:38909 I0408 18:33:39.683645 8120 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:39.683666 8120 webserver.cc:184] Document root disabled I0408 18:33:39.683888 8120 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:40235/ I0408 18:33:39.684113 8120 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:39.684558 8179 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:39.686650 8179 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3: Bootstrap starting. I0408 18:33:39.687070 8179 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3: No blocks or log segments found. Creating new log. I0408 18:33:39.687309 8179 log.cc:520] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3: Log is configured to *not* fsync() on all Append() calls I0408 18:33:39.687762 8179 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3: No bootstrap required, opened a new log I0408 18:33:39.688992 8179 raft_consensus.cc:304] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6499991a1c95436295562bcb3be311e3" member_type: VOTER } I0408 18:33:39.689038 8179 raft_consensus.cc:330] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:39.689059 8179 raft_consensus.cc:605] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6499991a1c95436295562bcb3be311e3, State: Initialized, Role: FOLLOWER I0408 18:33:39.689167 8179 consensus_queue.cc:226] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6499991a1c95436295562bcb3be311e3" member_type: VOTER } I0408 18:33:39.689190 8179 raft_consensus.cc:344] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:39.689205 8179 raft_consensus.cc:436] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:39.689230 8179 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3: Snoozing failure detection for 1.828s (starting election) I0408 18:33:39.689247 8179 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:39.689668 8179 raft_consensus.cc:458] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6499991a1c95436295562bcb3be311e3" member_type: VOTER } I0408 18:33:39.689731 8179 leader_election.cc:258] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:39.689888 8179 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6499991a1c95436295562bcb3be311e3" member_type: VOTER } } I0408 18:33:39.689914 8179 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [sys.catalog]: This master's current role is: FOLLOWER I0408 18:33:39.690006 8179 sys_catalog.cc:422] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:39.690346 8120 master_main.cc:79] Master server successfully started. I0408 18:33:39.690134 8181 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3: Snoozing failure detection for 2.248s (election complete) I0408 18:33:39.690374 8181 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:39.690490 8181 raft_consensus.cc:572] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [term 1 LEADER]: Becoming Leader. State: Replica: 6499991a1c95436295562bcb3be311e3, State: Running, Role: LEADER I0408 18:33:39.690558 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 8120 I0408 18:33:39.690579 8181 consensus_queue.cc:203] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6499991a1c95436295562bcb3be311e3" member_type: VOTER } I0408 18:33:39.690768 8181 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6499991a1c95436295562bcb3be311e3. Latest consensus state: current_term: 1 leader_uuid: "6499991a1c95436295562bcb3be311e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6499991a1c95436295562bcb3be311e3" member_type: VOTER } } I0408 18:33:39.690830 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:38909 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --fault_crash_during_log_replay=0.05 with env {} I0408 18:33:39.691187 8181 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [sys.catalog]: This master's current role is: LEADER W0408 18:33:39.691200 8183 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:39.691319 8183 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:39.690814 8182 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6499991a1c95436295562bcb3be311e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6499991a1c95436295562bcb3be311e3" member_type: VOTER } } I0408 18:33:39.691366 8182 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3 [sys.catalog]: This master's current role is: LEADER I0408 18:33:39.691440 8187 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:39.691603 8187 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0408 18:33:39.727449 8187 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:39.728125 8187 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:39.733886 8187 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 6499991a1c95436295562bcb3be311e3: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:39.752697 8188 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:39.752874 8188 flags.cc:412] Enabled unsafe flag: --fault_crash_during_log_replay=0.050000000000000003 W0408 18:33:39.752883 8188 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:39.753887 8188 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:39.753924 8188 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:39.755340 8188 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --fault_crash_during_log_replay=0.050000000000000003 --tserver_master_addrs=127.6.54.62:38909 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.8188 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:39.756759 8188 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:39.757053 8188 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28511us I0408 18:33:39.757238 8188 fs_manager.cc:260] Metadata directory not provided I0408 18:33:39.757279 8188 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:39.757366 8188 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:39.757390 8188 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:39.759284 8188 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.759505 8188 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.763442 8188 fs_manager.cc:495] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s I0408 18:33:39.763823 8188 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:39.763865 8188 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:39.763978 8188 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:39.764015 8188 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.764061 8188 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:39.826603 8188 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:39.827018 8188 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:39.827113 8188 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:39.831667 8188 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:37985 I0408 18:33:39.831739 8188 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:39.831760 8188 webserver.cc:184] Document root disabled I0408 18:33:39.831975 8188 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:41725/ I0408 18:33:39.832196 8188 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:39.832674 8188 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:39.833693 8310 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:39.833711 8310 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:39.833717 8310 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:39.838088 8309 heartbeater.cc:346] Connected to a master server at 127.6.54.62:38909 I0408 18:33:39.838140 8309 heartbeater.cc:426] Registering TS with master... I0408 18:33:39.838230 8309 heartbeater.cc:475] Master 127.6.54.62:38909 requested a full tablet report, sending... I0408 18:33:39.838373 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 8188 I0408 18:33:39.838683 8142 ts_manager.cc:81] Registered new tserver with Master: 4963d6fafa9d4dedba3c3ed40cd806a3 (127.6.54.1:37985) I0408 18:33:39.839586 8142 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:49011 I0408 18:33:39.842051 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:39.846158 6360 test_util.cc:200] Using random seed: 323465230 I0408 18:33:39.850719 8142 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39446: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:39.855239 8246 tablet_service.cc:758] Processing CreateTablet for tablet efa5dd7500824daa9a10776c7b78fd2e (table=test-workload [id=9cbf967898204309986a300220e3d691]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:39.855396 8246 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet efa5dd7500824daa9a10776c7b78fd2e. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:39.857053 8246 ts_tablet_manager.cc:1080] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:39.857183 8326 ts_tablet_manager.cc:938] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Bootstrapping tablet I0408 18:33:39.857259 8326 tablet_bootstrap.cc:436] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Bootstrap starting. I0408 18:33:39.857712 8326 tablet_bootstrap.cc:581] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: No blocks or log segments found. Creating new log. I0408 18:33:39.857945 8326 log.cc:520] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Log is configured to *not* fsync() on all Append() calls I0408 18:33:39.858342 8326 tablet_bootstrap.cc:436] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: No bootstrap required, opened a new log I0408 18:33:39.858398 8326 ts_tablet_manager.cc:955] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:39.859493 8326 raft_consensus.cc:304] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } } I0408 18:33:39.859542 8326 raft_consensus.cc:330] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:39.859557 8326 raft_consensus.cc:605] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4963d6fafa9d4dedba3c3ed40cd806a3, State: Initialized, Role: FOLLOWER I0408 18:33:39.859642 8326 consensus_queue.cc:226] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } } I0408 18:33:39.859666 8326 raft_consensus.cc:344] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:39.859676 8326 raft_consensus.cc:436] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:39.859692 8326 raft_consensus.cc:2700] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Snoozing failure detection for 1.513s (starting election) I0408 18:33:39.859707 8326 raft_consensus.cc:2752] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:39.860105 8326 raft_consensus.cc:458] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } } I0408 18:33:39.860157 8326 leader_election.cc:258] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:39.860288 8328 raft_consensus.cc:2700] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Snoozing failure detection for 1.601s (election complete) I0408 18:33:39.860306 8328 raft_consensus.cc:2507] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:39.860342 8309 heartbeater.cc:467] Master 127.6.54.62:38909 was elected leader, sending a full tablet report... I0408 18:33:39.860385 8328 raft_consensus.cc:572] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 1 LEADER]: Becoming Leader. State: Replica: 4963d6fafa9d4dedba3c3ed40cd806a3, State: Running, Role: LEADER I0408 18:33:39.860348 8326 ts_tablet_manager.cc:983] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:39.860464 8328 consensus_queue.cc:203] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } } I0408 18:33:39.861122 8142 catalog_manager.cc:3595] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 reported cstate change: term changed from 0 to 1, leader changed from to 4963d6fafa9d4dedba3c3ed40cd806a3 (127.6.54.1). New cstate: current_term: 1 leader_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } health_report { overall_health: HEALTHY } } } I0408 18:33:39.916481 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 8188 W0408 18:33:39.919493 8322 connection.cc:513] client connection to 127.6.54.1:37985 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.919565 8129 connection.cc:513] server connection from 127.6.54.1:49011 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.919661 8131 connection.cc:513] client connection to 127.6.54.1:37985 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:39.919834 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:37985 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=41725 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:38909 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --fault_crash_during_log_replay=0.05 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:39.952232 8342 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:39.952368 8342 flags.cc:412] Enabled unsafe flag: --fault_crash_during_log_replay=0.050000000000000003 W0408 18:33:39.952373 8342 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:39.953047 8342 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:39.953071 8342 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:39.954028 8342 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:37985 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=41725 --fault_crash_during_log_replay=0.050000000000000003 --tserver_master_addrs=127.6.54.62:38909 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.8342 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:39.954998 8342 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:39.955317 8342 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 28511us I0408 18:33:39.955448 8342 fs_manager.cc:260] Metadata directory not provided I0408 18:33:39.955479 8342 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:39.956492 8342 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:39.956724 8342 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:39.956758 8342 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:39.956861 8342 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:39.957267 8342 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:39.957319 8342 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:39.983228 8342 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:39.983480 8342 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:39.983500 8342 ts_tablet_manager.cc:915] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Loading tablet metadata I0408 18:33:39.984238 8342 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:39.984784 8342 ts_tablet_manager.cc:1080] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:39.984930 8342 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:39.984936 8357 ts_tablet_manager.cc:938] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Bootstrapping tablet I0408 18:33:39.985052 8357 tablet_bootstrap.cc:436] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Bootstrap starting. I0408 18:33:39.990461 8342 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:37985 I0408 18:33:39.990569 8342 webserver.cc:173] Starting webserver on 127.6.54.1:41725 I0408 18:33:39.990594 8342 webserver.cc:184] Document root disabled I0408 18:33:39.991014 8342 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:41725/ I0408 18:33:39.991328 8342 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:39.991690 8342 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:39.991706 8463 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:39.991720 8463 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:39.991725 8463 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:39.992696 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 8342 I0408 18:33:39.995004 8357 log.cc:520] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Log is configured to *not* fsync() on all Append() calls I0408 18:33:39.995678 8462 heartbeater.cc:346] Connected to a master server at 127.6.54.62:38909 I0408 18:33:39.995719 8462 heartbeater.cc:426] Registering TS with master... I0408 18:33:39.995810 8462 heartbeater.cc:475] Master 127.6.54.62:38909 requested a full tablet report, sending... W0408 18:33:39.995873 8357 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:39.995893 8357 fault_injection.cc:42] THIS SERVER MAY CRASH! E0408 18:33:39.996047 8357 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_during_log_replay (process will exit) I0408 18:33:39.996284 8138 ts_manager.cc:87] Re-registered known tserver with Master: 4963d6fafa9d4dedba3c3ed40cd806a3 (127.6.54.1:37985) I0408 18:33:39.997035 8138 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:33689 W0408 18:33:39.999099 8129 connection.cc:513] server connection from 127.6.54.1:33689 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:39.999323 8129 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0408 18:33:40.003108 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:37985 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=41725 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:38909 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:40.037995 8466 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:40.038240 8466 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:40.038930 8466 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:40.038960 8466 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:40.040045 8466 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:37985 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=41725 --tserver_master_addrs=127.6.54.62:38909 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.8466 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:40.041134 8466 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:40.041369 8466 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 29011us I0408 18:33:40.041512 8466 fs_manager.cc:260] Metadata directory not provided I0408 18:33:40.041559 8466 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:40.042650 8466 fs_manager.cc:397] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I0408 18:33:40.042910 8466 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:40.042943 8466 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:40.043031 8466 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:40.043473 8466 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" format_stamp: "Formatted at 2019-04-08 18:33:39 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.043520 8466 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:40.057121 8466 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:40.057343 8466 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:40.057365 8466 ts_tablet_manager.cc:915] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Loading tablet metadata I0408 18:33:40.058022 8466 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:40.058540 8466 ts_tablet_manager.cc:1080] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:40.058692 8466 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:40.058698 8481 ts_tablet_manager.cc:938] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Bootstrapping tablet I0408 18:33:40.058842 8481 tablet_bootstrap.cc:436] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Bootstrap starting. I0408 18:33:40.059294 8481 tablet_bootstrap.cc:645] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Previous recovery directory found at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/efa5dd7500824daa9a10776c7b78fd2e.recovery: Replaying log files from this location instead of /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/efa5dd7500824daa9a10776c7b78fd2e I0408 18:33:40.059317 8481 tablet_bootstrap.cc:652] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Deleting old log files from previous recovery attempt in /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/efa5dd7500824daa9a10776c7b78fd2e I0408 18:33:40.064515 8466 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:37985 I0408 18:33:40.064651 8466 webserver.cc:173] Starting webserver on 127.6.54.1:41725 I0408 18:33:40.064672 8466 webserver.cc:184] Document root disabled I0408 18:33:40.064965 8466 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:41725/ I0408 18:33:40.065263 8466 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:40.065798 8466 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:40.065959 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 8466 I0408 18:33:40.066164 8587 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:40.066184 8587 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:40.066190 8587 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:40.068153 8481 log.cc:520] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Log is configured to *not* fsync() on all Append() calls I0408 18:33:40.070502 8586 heartbeater.cc:346] Connected to a master server at 127.6.54.62:38909 I0408 18:33:40.070564 8586 heartbeater.cc:426] Registering TS with master... I0408 18:33:40.070662 8586 heartbeater.cc:475] Master 127.6.54.62:38909 requested a full tablet report, sending... I0408 18:33:40.071074 8140 ts_manager.cc:87] Re-registered known tserver with Master: 4963d6fafa9d4dedba3c3ed40cd806a3 (127.6.54.1:37985) I0408 18:33:40.071729 8140 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:39005 I0408 18:33:40.080543 8481 tablet_bootstrap.cc:436] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=228 overwritten=0 applied=228 ignored=0} inserts{seen=227 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0408 18:33:40.080749 8481 tablet_bootstrap.cc:436] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Bootstrap complete. I0408 18:33:40.081446 8481 ts_tablet_manager.cc:955] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.009s I0408 18:33:40.082497 8481 raft_consensus.cc:304] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } } I0408 18:33:40.082552 8481 raft_consensus.cc:605] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4963d6fafa9d4dedba3c3ed40cd806a3, State: Initialized, Role: FOLLOWER I0408 18:33:40.082659 8481 consensus_queue.cc:226] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 228, Last appended: 1.228, Last appended by leader: 228, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } } I0408 18:33:40.082684 8481 raft_consensus.cc:344] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:40.082705 8481 raft_consensus.cc:436] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:40.082728 8481 raft_consensus.cc:2700] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Snoozing failure detection for 2.215s (starting election) I0408 18:33:40.082749 8481 raft_consensus.cc:2752] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:40.083441 8481 raft_consensus.cc:458] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } } I0408 18:33:40.083495 8481 leader_election.cc:258] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:40.083681 8598 raft_consensus.cc:2700] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Snoozing failure detection for 1.715s (election complete) I0408 18:33:40.083711 8598 raft_consensus.cc:2507] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:40.083760 8481 ts_tablet_manager.cc:983] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s I0408 18:33:40.083781 8598 raft_consensus.cc:572] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [term 2 LEADER]: Becoming Leader. State: Replica: 4963d6fafa9d4dedba3c3ed40cd806a3, State: Running, Role: LEADER I0408 18:33:40.083811 8586 heartbeater.cc:467] Master 127.6.54.62:38909 was elected leader, sending a full tablet report... I0408 18:33:40.083842 8598 consensus_queue.cc:203] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 1.228, Last appended by leader: 228, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } } I0408 18:33:40.084309 8140 catalog_manager.cc:3595] T efa5dd7500824daa9a10776c7b78fd2e P 4963d6fafa9d4dedba3c3ed40cd806a3 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4963d6fafa9d4dedba3c3ed40cd806a3" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 37985 } health_report { overall_health: HEALTHY } } } I0408 18:33:40.277449 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 8466 W0408 18:33:40.280604 8128 connection.cc:513] server connection from 127.6.54.1:39005 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:40.280707 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 8120 W0408 18:33:40.283195 8317 connection.cc:513] client connection to 127.6.54.62:38909 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:40.283207 8119 connection.cc:513] client connection to 127.6.54.62:38909 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestCrashDuringLogReplay/0 (678 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestCrashDuringLogReplay/1 I0408 18:33:40.285667 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:40.318615 8604 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:40.318768 8604 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:40.319514 8604 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:40.319537 8604 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:40.319541 8604 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:40.320571 8604 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.8604 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:40.321471 8604 master_main.cc:73] Initializing master server... I0408 18:33:40.321687 8604 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 29011us I0408 18:33:40.321871 8604 fs_manager.cc:260] Metadata directory not provided I0408 18:33:40.321904 8604 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:40.321985 8604 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:40.322002 8604 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:40.323335 8604 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "94896d5610134cb5b3bc7cea3f7f52ac" format_stamp: "Formatted at 2019-04-08 18:33:40 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.323511 8604 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "94896d5610134cb5b3bc7cea3f7f52ac" format_stamp: "Formatted at 2019-04-08 18:33:40 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.324733 8604 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s I0408 18:33:40.324995 8604 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:40.325024 8604 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:40.325644 8604 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:40.325676 8604 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/wal uuid: "94896d5610134cb5b3bc7cea3f7f52ac" format_stamp: "Formatted at 2019-04-08 18:33:40 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.325744 8604 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:40.407444 8604 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:40.407516 8604 master_main.cc:76] Starting Master server... I0408 18:33:40.407665 8619 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:40.407686 8619 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:40.407690 8619 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:40.409790 8604 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:35087 I0408 18:33:40.409852 8604 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:40.409878 8604 webserver.cc:184] Document root disabled I0408 18:33:40.410095 8604 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:46203/ I0408 18:33:40.410307 8604 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:40.410871 8663 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:40.413022 8663 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac: Bootstrap starting. I0408 18:33:40.413465 8663 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac: No blocks or log segments found. Creating new log. I0408 18:33:40.413717 8663 log.cc:520] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac: Log is configured to *not* fsync() on all Append() calls I0408 18:33:40.414120 8663 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac: No bootstrap required, opened a new log I0408 18:33:40.415359 8663 raft_consensus.cc:304] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" member_type: VOTER } I0408 18:33:40.415396 8663 raft_consensus.cc:330] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:40.415410 8663 raft_consensus.cc:605] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94896d5610134cb5b3bc7cea3f7f52ac, State: Initialized, Role: FOLLOWER I0408 18:33:40.415521 8663 consensus_queue.cc:226] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" member_type: VOTER } I0408 18:33:40.415552 8663 raft_consensus.cc:344] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:40.415565 8663 raft_consensus.cc:436] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:40.415593 8663 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac: Snoozing failure detection for 2.058s (starting election) I0408 18:33:40.415612 8663 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:40.416021 8663 raft_consensus.cc:458] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" member_type: VOTER } I0408 18:33:40.416079 8663 leader_election.cc:258] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:40.416261 8663 sys_catalog.cc:335] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" member_type: VOTER } } I0408 18:33:40.416286 8663 sys_catalog.cc:338] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [sys.catalog]: This master's current role is: FOLLOWER I0408 18:33:40.416383 8663 sys_catalog.cc:422] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:40.416611 8604 master_main.cc:79] Master server successfully started. I0408 18:33:40.416810 8665 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac: Snoozing failure detection for 2.230s (election complete) I0408 18:33:40.416836 8665 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:40.416908 8665 raft_consensus.cc:572] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [term 1 LEADER]: Becoming Leader. State: Replica: 94896d5610134cb5b3bc7cea3f7f52ac, State: Running, Role: LEADER I0408 18:33:40.416966 8665 consensus_queue.cc:203] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" member_type: VOTER } I0408 18:33:40.417145 8665 sys_catalog.cc:335] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94896d5610134cb5b3bc7cea3f7f52ac. Latest consensus state: current_term: 1 leader_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" member_type: VOTER } } I0408 18:33:40.417163 8665 sys_catalog.cc:338] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [sys.catalog]: This master's current role is: LEADER I0408 18:33:40.417201 8666 sys_catalog.cc:335] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94896d5610134cb5b3bc7cea3f7f52ac" member_type: VOTER } } I0408 18:33:40.417222 8666 sys_catalog.cc:338] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac [sys.catalog]: This master's current role is: LEADER W0408 18:33:40.417501 8667 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:40.417541 8667 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:40.417667 8670 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:40.417796 8670 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0408 18:33:40.418882 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 8604 I0408 18:33:40.419116 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:35087 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --fault_crash_during_log_replay=0.05 with env {} I0408 18:33:40.438273 8670 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:40.439774 8670 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:40.459022 8670 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 94896d5610134cb5b3bc7cea3f7f52ac: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:40.468277 8672 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:40.468410 8672 flags.cc:412] Enabled unsafe flag: --fault_crash_during_log_replay=0.050000000000000003 W0408 18:33:40.468415 8672 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:40.469112 8672 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:40.469139 8672 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:40.470147 8672 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --fault_crash_during_log_replay=0.050000000000000003 --tserver_master_addrs=127.6.54.62:35087 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.8672 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:40.471094 8672 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:40.471349 8672 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 29011us I0408 18:33:40.471487 8672 fs_manager.cc:260] Metadata directory not provided I0408 18:33:40.471514 8672 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:40.471602 8672 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:40.471621 8672 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:40.472879 8672 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "70c384ea30ac4de89fc4758a52640458" format_stamp: "Formatted at 2019-04-08 18:33:40 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.473052 8672 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "70c384ea30ac4de89fc4758a52640458" format_stamp: "Formatted at 2019-04-08 18:33:40 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.474237 8672 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:40.474475 8672 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:40.474501 8672 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:40.475136 8672 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:40.475193 8672 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "70c384ea30ac4de89fc4758a52640458" format_stamp: "Formatted at 2019-04-08 18:33:40 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.475244 8672 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:40.490736 8672 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:40.490914 8672 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:40.490986 8672 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:40.495344 8672 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:42063 I0408 18:33:40.495450 8672 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:40.495467 8672 webserver.cc:184] Document root disabled I0408 18:33:40.495700 8672 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:44701/ I0408 18:33:40.495889 8672 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:40.496191 8672 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:40.496318 8794 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:40.496345 8794 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:40.496351 8794 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:40.501893 8793 heartbeater.cc:346] Connected to a master server at 127.6.54.62:35087 I0408 18:33:40.501945 8793 heartbeater.cc:426] Registering TS with master... I0408 18:33:40.502053 8793 heartbeater.cc:475] Master 127.6.54.62:35087 requested a full tablet report, sending... I0408 18:33:40.502522 8630 ts_manager.cc:81] Registered new tserver with Master: 70c384ea30ac4de89fc4758a52640458 (127.6.54.1:42063) I0408 18:33:40.503330 8630 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:42369 I0408 18:33:40.505054 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 8672 I0408 18:33:40.509644 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:40.513962 6360 test_util.cc:200] Using random seed: 324133033 I0408 18:33:40.518651 8630 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40032: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:40.526873 8730 tablet_service.cc:758] Processing CreateTablet for tablet f0d0f016d23f4453b8b7db8b91efca46 (table=test-workload [id=af9c87980b374e09aa4b79aa736d17db]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:40.527053 8730 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet f0d0f016d23f4453b8b7db8b91efca46. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:40.529647 8730 ts_tablet_manager.cc:1080] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:40.529876 8810 ts_tablet_manager.cc:938] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Bootstrapping tablet I0408 18:33:40.529955 8810 tablet_bootstrap.cc:436] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Bootstrap starting. I0408 18:33:40.530437 8810 tablet_bootstrap.cc:581] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: No blocks or log segments found. Creating new log. I0408 18:33:40.530738 8810 log.cc:520] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Log is configured to *not* fsync() on all Append() calls I0408 18:33:40.531262 8810 tablet_bootstrap.cc:436] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: No bootstrap required, opened a new log I0408 18:33:40.531323 8810 ts_tablet_manager.cc:955] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:40.532369 8810 raft_consensus.cc:304] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } } I0408 18:33:40.532413 8810 raft_consensus.cc:330] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:40.532435 8810 raft_consensus.cc:605] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c384ea30ac4de89fc4758a52640458, State: Initialized, Role: FOLLOWER I0408 18:33:40.532591 8810 consensus_queue.cc:226] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } } I0408 18:33:40.532629 8810 raft_consensus.cc:344] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:40.532644 8810 raft_consensus.cc:436] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:40.532671 8810 raft_consensus.cc:2700] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Snoozing failure detection for 2.220s (starting election) I0408 18:33:40.532686 8810 raft_consensus.cc:2752] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:40.533152 8810 raft_consensus.cc:458] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } } I0408 18:33:40.533213 8810 leader_election.cc:258] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:40.533406 8812 raft_consensus.cc:2700] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Snoozing failure detection for 2.042s (election complete) I0408 18:33:40.533430 8812 raft_consensus.cc:2507] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:40.533465 8793 heartbeater.cc:467] Master 127.6.54.62:35087 was elected leader, sending a full tablet report... I0408 18:33:40.533501 8812 raft_consensus.cc:572] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 1 LEADER]: Becoming Leader. State: Replica: 70c384ea30ac4de89fc4758a52640458, State: Running, Role: LEADER I0408 18:33:40.533582 8812 consensus_queue.cc:203] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } } I0408 18:33:40.533632 8810 ts_tablet_manager.cc:983] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I0408 18:33:40.534257 8630 catalog_manager.cc:3595] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 reported cstate change: term changed from 0 to 1, leader changed from to 70c384ea30ac4de89fc4758a52640458 (127.6.54.1). New cstate: current_term: 1 leader_uuid: "70c384ea30ac4de89fc4758a52640458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } health_report { overall_health: HEALTHY } } } I0408 18:33:40.588564 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 8672 W0408 18:33:40.591514 8803 connection.cc:513] client connection to 127.6.54.1:42063 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:40.591560 8614 connection.cc:513] server connection from 127.6.54.1:42369 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:40.591572 8612 connection.cc:513] client connection to 127.6.54.1:42063 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:40.591920 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:42063 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=44701 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:35087 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --fault_crash_during_log_replay=0.05 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:40.623994 8826 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:40.624132 8826 flags.cc:412] Enabled unsafe flag: --fault_crash_during_log_replay=0.050000000000000003 W0408 18:33:40.624138 8826 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:40.624833 8826 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:40.624857 8826 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:40.625823 8826 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:42063 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=44701 --fault_crash_during_log_replay=0.050000000000000003 --tserver_master_addrs=127.6.54.62:35087 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.8826 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:40.626781 8826 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:40.627012 8826 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 29011us I0408 18:33:40.627185 8826 fs_manager.cc:260] Metadata directory not provided I0408 18:33:40.627235 8826 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:40.628229 8826 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:40.628428 8826 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:40.628458 8826 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:40.629036 8826 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:40.629395 8826 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "70c384ea30ac4de89fc4758a52640458" format_stamp: "Formatted at 2019-04-08 18:33:40 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.629446 8826 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:40.662976 8826 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:40.663301 8826 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:40.663322 8826 ts_tablet_manager.cc:915] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Loading tablet metadata I0408 18:33:40.663918 8826 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:40.664412 8826 ts_tablet_manager.cc:1080] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:40.664556 8826 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:40.664558 8841 ts_tablet_manager.cc:938] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Bootstrapping tablet I0408 18:33:40.664710 8841 tablet_bootstrap.cc:436] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Bootstrap starting. I0408 18:33:40.669831 8826 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:42063 I0408 18:33:40.669909 8826 webserver.cc:173] Starting webserver on 127.6.54.1:44701 I0408 18:33:40.669924 8826 webserver.cc:184] Document root disabled I0408 18:33:40.670137 8826 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:44701/ I0408 18:33:40.670348 8826 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:40.670683 8826 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:40.670725 8947 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:40.670766 8947 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:40.670773 8947 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:40.674476 8841 log.cc:520] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Log is configured to *not* fsync() on all Append() calls I0408 18:33:40.674911 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 8826 I0408 18:33:40.675268 8946 heartbeater.cc:346] Connected to a master server at 127.6.54.62:35087 I0408 18:33:40.675316 8946 heartbeater.cc:426] Registering TS with master... W0408 18:33:40.675314 8841 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:40.675328 8841 fault_injection.cc:42] THIS SERVER MAY CRASH! I0408 18:33:40.675412 8946 heartbeater.cc:475] Master 127.6.54.62:35087 requested a full tablet report, sending... I0408 18:33:40.675917 8629 ts_manager.cc:87] Re-registered known tserver with Master: 70c384ea30ac4de89fc4758a52640458 (127.6.54.1:42063) I0408 18:33:40.676714 8629 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:36655 E0408 18:33:40.677243 8841 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_during_log_replay (process will exit) W0408 18:33:40.680147 8613 connection.cc:513] server connection from 127.6.54.1:36655 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:40.685286 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:42063 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=44701 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:35087 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:40.717800 8950 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:40.717936 8950 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:40.718597 8950 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:40.718618 8950 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:40.719569 8950 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:42063 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=44701 --tserver_master_addrs=127.6.54.62:35087 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.8950 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:40.720507 8950 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:40.720757 8950 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 29011us I0408 18:33:40.720896 8950 fs_manager.cc:260] Metadata directory not provided I0408 18:33:40.720933 8950 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:40.721916 8950 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:40.722151 8950 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:40.722177 8950 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:40.722784 8950 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:40.723139 8950 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "70c384ea30ac4de89fc4758a52640458" format_stamp: "Formatted at 2019-04-08 18:33:40 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:40.723212 8950 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:40.743748 8950 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:40.743942 8950 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:40.743963 8950 ts_tablet_manager.cc:915] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Loading tablet metadata I0408 18:33:40.744545 8950 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:40.745033 8950 ts_tablet_manager.cc:1080] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:40.745163 8965 ts_tablet_manager.cc:938] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Bootstrapping tablet I0408 18:33:40.745170 8950 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:40.745276 8965 tablet_bootstrap.cc:436] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Bootstrap starting. I0408 18:33:40.745703 8965 tablet_bootstrap.cc:645] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Previous recovery directory found at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/f0d0f016d23f4453b8b7db8b91efca46.recovery: Replaying log files from this location instead of /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/f0d0f016d23f4453b8b7db8b91efca46 I0408 18:33:40.745730 8965 tablet_bootstrap.cc:652] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Deleting old log files from previous recovery attempt in /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/f0d0f016d23f4453b8b7db8b91efca46 I0408 18:33:40.750627 8950 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:42063 I0408 18:33:40.750728 8950 webserver.cc:173] Starting webserver on 127.6.54.1:44701 I0408 18:33:40.750747 8950 webserver.cc:184] Document root disabled I0408 18:33:40.751065 8950 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:44701/ I0408 18:33:40.751407 8950 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashDuringLogReplay_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:40.751816 8950 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:40.752338 9071 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:40.752368 9071 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:40.752377 9071 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:40.753159 8965 log.cc:520] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Log is configured to *not* fsync() on all Append() calls I0408 18:33:40.755811 9070 heartbeater.cc:346] Connected to a master server at 127.6.54.62:35087 I0408 18:33:40.755861 9070 heartbeater.cc:426] Registering TS with master... I0408 18:33:40.755939 9070 heartbeater.cc:475] Master 127.6.54.62:35087 requested a full tablet report, sending... I0408 18:33:40.756698 8629 ts_manager.cc:87] Re-registered known tserver with Master: 70c384ea30ac4de89fc4758a52640458 (127.6.54.1:42063) I0408 18:33:40.757195 8629 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:49419 I0408 18:33:40.758169 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 8950 I0408 18:33:40.767729 8965 tablet_bootstrap.cc:436] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Bootstrap replayed 1/1 log segments. Stats: ops{read=232 overwritten=0 applied=232 ignored=0} inserts{seen=231 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0408 18:33:40.767941 8965 tablet_bootstrap.cc:436] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Bootstrap complete. I0408 18:33:40.768664 8965 ts_tablet_manager.cc:955] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.008s I0408 18:33:40.769780 8965 raft_consensus.cc:304] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } } I0408 18:33:40.769829 8965 raft_consensus.cc:605] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c384ea30ac4de89fc4758a52640458, State: Initialized, Role: FOLLOWER I0408 18:33:40.769937 8965 consensus_queue.cc:226] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 232, Last appended: 1.232, Last appended by leader: 232, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } } I0408 18:33:40.769961 8965 raft_consensus.cc:344] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:40.769984 8965 raft_consensus.cc:436] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:40.770017 8965 raft_consensus.cc:2700] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Snoozing failure detection for 1.657s (starting election) I0408 18:33:40.770035 8965 raft_consensus.cc:2752] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:40.770678 8965 raft_consensus.cc:458] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } } I0408 18:33:40.770735 8965 leader_election.cc:258] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:40.770910 9070 heartbeater.cc:467] Master 127.6.54.62:35087 was elected leader, sending a full tablet report... I0408 18:33:40.770917 8965 ts_tablet_manager.cc:983] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s I0408 18:33:40.770965 9081 raft_consensus.cc:2700] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458: Snoozing failure detection for 1.889s (election complete) I0408 18:33:40.770982 9081 raft_consensus.cc:2507] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:40.771046 9081 raft_consensus.cc:572] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [term 2 LEADER]: Becoming Leader. State: Replica: 70c384ea30ac4de89fc4758a52640458, State: Running, Role: LEADER I0408 18:33:40.771121 9081 consensus_queue.cc:203] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 1.232, Last appended by leader: 232, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } } I0408 18:33:40.771903 8629 catalog_manager.cc:3595] T f0d0f016d23f4453b8b7db8b91efca46 P 70c384ea30ac4de89fc4758a52640458 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "70c384ea30ac4de89fc4758a52640458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c384ea30ac4de89fc4758a52640458" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 42063 } health_report { overall_health: HEALTHY } } } I0408 18:33:40.967934 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 8950 W0408 18:33:40.970926 8615 connection.cc:513] server connection from 127.6.54.1:49419 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:40.970989 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 8604 W0408 18:33:40.973587 8799 connection.cc:513] client connection to 127.6.54.62:35087 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:40.973620 8603 connection.cc:513] client connection to 127.6.54.62:35087 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestCrashDuringLogReplay/1 (690 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader/0 I0408 18:33:40.976150 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data --block_manager=file --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:41.011418 9087 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:41.011597 9087 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:41.012416 9087 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:41.012434 9087 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:41.012439 9087 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:41.013607 9087 master_main.cc:66] Master server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.9087 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:41.014466 9087 master_main.cc:73] Initializing master server... I0408 18:33:41.014688 9087 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 29511us I0408 18:33:41.014971 9087 fs_manager.cc:260] Metadata directory not provided I0408 18:33:41.015007 9087 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:41.015085 9087 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:41.015105 9087 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:41.016495 9087 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "043696b794e74b698e2c618241415087" format_stamp: "Formatted at 2019-04-08 18:33:41 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:41.016685 9087 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "043696b794e74b698e2c618241415087" format_stamp: "Formatted at 2019-04-08 18:33:41 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:41.017840 9087 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0408 18:33:41.018085 9087 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:41.018119 9087 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:41.018210 9087 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:41.018241 9087 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/wal uuid: "043696b794e74b698e2c618241415087" format_stamp: "Formatted at 2019-04-08 18:33:41 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:41.018276 9087 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:41.049199 9087 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:41.049289 9087 master_main.cc:76] Starting Master server... I0408 18:33:41.049435 9102 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:41.049460 9102 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:41.049466 9102 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:41.051795 9087 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:36737 I0408 18:33:41.051872 9087 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:41.051885 9087 webserver.cc:184] Document root disabled I0408 18:33:41.052094 9087 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:37861/ I0408 18:33:41.052312 9087 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:41.052779 9146 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:41.054913 9146 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087: Bootstrap starting. I0408 18:33:41.055399 9146 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087: No blocks or log segments found. Creating new log. I0408 18:33:41.055647 9146 log.cc:520] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087: Log is configured to *not* fsync() on all Append() calls I0408 18:33:41.056032 9146 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087: No bootstrap required, opened a new log I0408 18:33:41.057195 9146 raft_consensus.cc:304] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043696b794e74b698e2c618241415087" member_type: VOTER } I0408 18:33:41.057231 9146 raft_consensus.cc:330] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:41.057245 9146 raft_consensus.cc:605] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043696b794e74b698e2c618241415087, State: Initialized, Role: FOLLOWER I0408 18:33:41.057356 9146 consensus_queue.cc:226] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043696b794e74b698e2c618241415087" member_type: VOTER } I0408 18:33:41.057376 9146 raft_consensus.cc:344] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:41.057387 9146 raft_consensus.cc:436] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:41.057407 9146 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087: Snoozing failure detection for 1.969s (starting election) I0408 18:33:41.057426 9146 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:41.057826 9146 raft_consensus.cc:458] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043696b794e74b698e2c618241415087" member_type: VOTER } I0408 18:33:41.057878 9146 leader_election.cc:258] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:41.058068 9148 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087: Snoozing failure detection for 1.628s (election complete) I0408 18:33:41.058095 9148 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:41.058174 9148 raft_consensus.cc:572] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [term 1 LEADER]: Becoming Leader. State: Replica: 043696b794e74b698e2c618241415087, State: Running, Role: LEADER I0408 18:33:41.058246 9148 consensus_queue.cc:203] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043696b794e74b698e2c618241415087" member_type: VOTER } I0408 18:33:41.058439 9148 sys_catalog.cc:335] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "043696b794e74b698e2c618241415087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043696b794e74b698e2c618241415087" member_type: VOTER } } I0408 18:33:41.058444 9146 sys_catalog.cc:335] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "043696b794e74b698e2c618241415087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043696b794e74b698e2c618241415087" member_type: VOTER } } I0408 18:33:41.058470 9146 sys_catalog.cc:338] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [sys.catalog]: This master's current role is: LEADER I0408 18:33:41.058523 9149 sys_catalog.cc:335] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043696b794e74b698e2c618241415087. Latest consensus state: current_term: 1 leader_uuid: "043696b794e74b698e2c618241415087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043696b794e74b698e2c618241415087" member_type: VOTER } } I0408 18:33:41.058550 9149 sys_catalog.cc:338] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [sys.catalog]: This master's current role is: LEADER I0408 18:33:41.058637 9146 sys_catalog.cc:422] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:41.058470 9148 sys_catalog.cc:338] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087 [sys.catalog]: This master's current role is: LEADER I0408 18:33:41.058868 9087 master_main.cc:79] Master server successfully started. I0408 18:33:41.059058 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 9087 W0408 18:33:41.059113 9153 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:41.059135 9153 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:41.059345 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36737 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_segment_size_mb=1 --log_compression_codec=NO_COMPRESSION with env {} I0408 18:33:41.063802 9152 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:41.063954 9152 catalog_manager.cc:983] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:41.108952 9155 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:41.109146 9155 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:41.110159 9155 flags.cc:412] Enabled experimental flag: --log_compression_codec=NO_COMPRESSION W0408 18:33:41.110185 9155 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:41.110204 9155 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:41.111627 9155 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=NO_COMPRESSION --log_segment_size_mb=1 --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36737 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.9155 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:41.112937 9155 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:41.113234 9155 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 29511us I0408 18:33:41.113418 9155 fs_manager.cc:260] Metadata directory not provided I0408 18:33:41.113461 9155 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:41.113579 9155 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:41.113605 9155 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:41.115414 9155 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "8dd0dee0c0eb412fb08f498e38e90e86" format_stamp: "Formatted at 2019-04-08 18:33:41 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:41.115461 9152 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:41.115669 9155 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "8dd0dee0c0eb412fb08f498e38e90e86" format_stamp: "Formatted at 2019-04-08 18:33:41 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:41.116117 9152 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:41.119434 9155 fs_manager.cc:495] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s I0408 18:33:41.119799 9155 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:41.119839 9155 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:41.119935 9155 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:41.119972 9155 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "8dd0dee0c0eb412fb08f498e38e90e86" format_stamp: "Formatted at 2019-04-08 18:33:41 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:41.120014 9155 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:41.123098 9152 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 043696b794e74b698e2c618241415087: Generated new TSK 0 I0408 18:33:41.158442 9155 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:41.158751 9155 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:41.158845 9155 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:41.163831 9155 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:39775 I0408 18:33:41.163905 9155 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:41.163920 9155 webserver.cc:184] Document root disabled I0408 18:33:41.164130 9155 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:38757/ I0408 18:33:41.164352 9155 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:41.164639 9155 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:41.165822 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 9155 I0408 18:33:41.167587 9277 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:41.167609 9277 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:41.167616 9277 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:41.172471 9276 heartbeater.cc:346] Connected to a master server at 127.6.54.62:36737 I0408 18:33:41.172511 9276 heartbeater.cc:426] Registering TS with master... I0408 18:33:41.172616 9276 heartbeater.cc:475] Master 127.6.54.62:36737 requested a full tablet report, sending... I0408 18:33:41.173058 9112 ts_manager.cc:81] Registered new tserver with Master: 8dd0dee0c0eb412fb08f498e38e90e86 (127.6.54.1:39775) I0408 18:33:41.173871 9112 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:46161 I0408 18:33:41.174641 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:41.182315 6360 test_util.cc:200] Using random seed: 324801387 I0408 18:33:41.187434 9112 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54752: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:41.194171 9207 tablet_service.cc:758] Processing CreateTablet for tablet 13d6aa5379f0483995aac68de5c7f473 (table=test-workload [id=e5ada3153b4e4030bbd1561d986000ef]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:41.194322 9207 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 13d6aa5379f0483995aac68de5c7f473. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:41.196173 9207 ts_tablet_manager.cc:1080] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:41.199215 9294 ts_tablet_manager.cc:938] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Bootstrapping tablet I0408 18:33:41.199288 9294 tablet_bootstrap.cc:436] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Bootstrap starting. I0408 18:33:41.199688 9294 tablet_bootstrap.cc:581] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: No blocks or log segments found. Creating new log. W0408 18:33:41.199874 9294 compression_codec.cc:281] Unable to recognize the compression codec 'NO_COMPRESSION' using no compression as default. I0408 18:33:41.199975 9294 log.cc:520] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Log is configured to *not* fsync() on all Append() calls I0408 18:33:41.200362 9294 tablet_bootstrap.cc:436] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: No bootstrap required, opened a new log I0408 18:33:41.200412 9294 ts_tablet_manager.cc:955] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:41.201488 9294 raft_consensus.cc:304] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } } I0408 18:33:41.201539 9294 raft_consensus.cc:330] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:41.201561 9294 raft_consensus.cc:605] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd0dee0c0eb412fb08f498e38e90e86, State: Initialized, Role: FOLLOWER I0408 18:33:41.201746 9294 consensus_queue.cc:226] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } } I0408 18:33:41.201774 9294 raft_consensus.cc:344] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:41.201789 9294 raft_consensus.cc:436] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:41.201814 9294 raft_consensus.cc:2700] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Snoozing failure detection for 2.007s (starting election) I0408 18:33:41.201833 9294 raft_consensus.cc:2752] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:41.202248 9294 raft_consensus.cc:458] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } } I0408 18:33:41.202308 9294 leader_election.cc:258] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:41.202497 9276 heartbeater.cc:467] Master 127.6.54.62:36737 was elected leader, sending a full tablet report... I0408 18:33:41.202512 9294 ts_tablet_manager.cc:983] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:41.203222 9296 raft_consensus.cc:2700] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Snoozing failure detection for 2.240s (election complete) I0408 18:33:41.203249 9296 raft_consensus.cc:2507] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:41.203322 9296 raft_consensus.cc:572] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 1 LEADER]: Becoming Leader. State: Replica: 8dd0dee0c0eb412fb08f498e38e90e86, State: Running, Role: LEADER I0408 18:33:41.203393 9296 consensus_queue.cc:203] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } } I0408 18:33:41.203999 9112 catalog_manager.cc:3595] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 reported cstate change: term changed from 0 to 1, leader changed from to 8dd0dee0c0eb412fb08f498e38e90e86 (127.6.54.1). New cstate: current_term: 1 leader_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } health_report { overall_health: HEALTHY } } } W0408 18:33:41.208546 9273 generic_service.cc:126] {username='slave'} at 127.0.0.1:33342 forcing change of non-runtime-safe flag fault_crash_before_write_log_segment_header I0408 18:33:41.208595 9273 generic_service.cc:146] {username='slave'} at 127.0.0.1:33342 changed flags via RPC: fault_crash_before_write_log_segment_header from '0' to '0.9' I0408 18:33:41.257309 9299 log.cc:644] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Max segment size reached. Starting new segment allocation W0408 18:33:41.258635 9299 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:41.258658 9299 fault_injection.cc:42] THIS SERVER MAY CRASH! E0408 18:33:41.258666 9299 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_write_log_segment_header (process will exit) W0408 18:33:41.262419 9290 connection.cc:659] client connection to 127.6.54.1:39775 send error: Network error: failed to write to TLS socket: Connection reset by peer W0408 18:33:41.262449 9290 meta_cache.cc:219] Tablet 13d6aa5379f0483995aac68de5c7f473: Replica 8dd0dee0c0eb412fb08f498e38e90e86 (127.6.54.1:39775) has failed: Network error: failed to write to TLS socket: Connection reset by peer W0408 18:33:41.262511 9086 connection.cc:513] client connection to 127.6.54.1:39775 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:41.262552 9290 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Success W0408 18:33:41.262538 9097 connection.cc:513] server connection from 127.6.54.1:46161 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:41.262646 9098 connection.cc:513] client connection to 127.6.54.1:39775 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:42.216732 9290 batcher.cc:349] Timed out: Failed to write batch of 50 ops to tablet 13d6aa5379f0483995aac68de5c7f473 after 40 attempt(s): Failed to write to server: (no server available): Write(tablet: 13d6aa5379f0483995aac68de5c7f473, num_ops: 50, num_attempts: 40) passed its deadline: Network error: failed to write to TLS socket: Connection reset by peer I0408 18:33:42.263550 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:39775 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=38757 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:36737 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_segment_size_mb=1 --log_compression_codec=NO_COMPRESSION with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:42.296274 9307 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:42.296417 9307 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:42.297107 9307 flags.cc:412] Enabled experimental flag: --log_compression_codec=NO_COMPRESSION W0408 18:33:42.297125 9307 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:42.297137 9307 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:42.298111 9307 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=NO_COMPRESSION --log_segment_size_mb=1 --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:39775 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=38757 --tserver_master_addrs=127.6.54.62:36737 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.9307 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:42.299060 9307 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:42.299321 9307 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 30011us I0408 18:33:42.299463 9307 fs_manager.cc:260] Metadata directory not provided I0408 18:33:42.299510 9307 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:42.300469 9307 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:42.300694 9307 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:42.300722 9307 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:42.300812 9307 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:42.301162 9307 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "8dd0dee0c0eb412fb08f498e38e90e86" format_stamp: "Formatted at 2019-04-08 18:33:41 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:42.301200 9307 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:42.325568 9307 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:42.325783 9307 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:42.325803 9307 ts_tablet_manager.cc:915] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Loading tablet metadata I0408 18:33:42.326334 9307 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:42.326856 9307 ts_tablet_manager.cc:1080] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:42.326990 9307 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:42.326990 9322 ts_tablet_manager.cc:938] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Bootstrapping tablet I0408 18:33:42.327105 9322 tablet_bootstrap.cc:436] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Bootstrap starting. W0408 18:33:42.328086 9322 log_util.cc:472] Log segment file /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/13d6aa5379f0483995aac68de5c7f473.recovery/wal-000000002 has 12 initial NULL bytes instead of magic and header length: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 and will be treated as a blank segment. W0408 18:33:42.328131 9322 log_reader.cc:149] Ignoring log segment wal-000000002 since it was uninitialized (probably left after a prior tablet server crash) W0408 18:33:42.328472 9322 compression_codec.cc:281] Unable to recognize the compression codec 'NO_COMPRESSION' using no compression as default. I0408 18:33:42.328572 9322 log.cc:520] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Log is configured to *not* fsync() on all Append() calls I0408 18:33:42.332563 9307 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:39775 I0408 18:33:42.332636 9307 webserver.cc:173] Starting webserver on 127.6.54.1:38757 I0408 18:33:42.332643 9307 webserver.cc:184] Document root disabled I0408 18:33:42.332834 9307 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:38757/ I0408 18:33:42.333012 9307 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:42.333575 9307 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:42.333724 9429 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:42.333743 9429 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:42.333748 9429 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:42.335430 9322 log.cc:644] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Max segment size reached. Starting new segment allocation I0408 18:33:42.335846 9322 tablet_bootstrap.cc:436] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0408 18:33:42.336091 9322 tablet_bootstrap.cc:436] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Bootstrap complete. I0408 18:33:42.336405 9322 ts_tablet_manager.cc:955] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s I0408 18:33:42.336797 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 9307 I0408 18:33:42.337625 9428 heartbeater.cc:346] Connected to a master server at 127.6.54.62:36737 I0408 18:33:42.337707 9428 heartbeater.cc:426] Registering TS with master... I0408 18:33:42.337800 9322 raft_consensus.cc:304] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } } I0408 18:33:42.337846 9428 heartbeater.cc:475] Master 127.6.54.62:36737 requested a full tablet report, sending... I0408 18:33:42.338244 9322 raft_consensus.cc:605] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd0dee0c0eb412fb08f498e38e90e86, State: Initialized, Role: FOLLOWER I0408 18:33:42.338377 9322 consensus_queue.cc:226] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } } I0408 18:33:42.338418 9322 raft_consensus.cc:344] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:42.338471 9322 raft_consensus.cc:436] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:42.338506 9322 raft_consensus.cc:2700] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Snoozing failure detection for 2.189s (starting election) I0408 18:33:42.338557 9322 raft_consensus.cc:2752] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:42.338878 9106 ts_manager.cc:87] Re-registered known tserver with Master: 8dd0dee0c0eb412fb08f498e38e90e86 (127.6.54.1:39775) I0408 18:33:42.339287 9322 raft_consensus.cc:458] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } } I0408 18:33:42.339342 9322 leader_election.cc:258] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:42.339519 9322 ts_tablet_manager.cc:983] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s I0408 18:33:42.339677 9106 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:53943 I0408 18:33:42.339946 9438 raft_consensus.cc:2700] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Snoozing failure detection for 1.840s (election complete) I0408 18:33:42.339977 9438 raft_consensus.cc:2507] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:42.340050 9438 raft_consensus.cc:572] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [term 2 LEADER]: Becoming Leader. State: Replica: 8dd0dee0c0eb412fb08f498e38e90e86, State: Running, Role: LEADER I0408 18:33:42.340135 9438 consensus_queue.cc:203] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } } I0408 18:33:42.340399 9428 heartbeater.cc:467] Master 127.6.54.62:36737 was elected leader, sending a full tablet report... I0408 18:33:42.340545 9441 log.cc:571] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/13d6aa5379f0483995aac68de5c7f473/wal-000000002 I0408 18:33:42.340770 9112 catalog_manager.cc:3595] T 13d6aa5379f0483995aac68de5c7f473 P 8dd0dee0c0eb412fb08f498e38e90e86 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd0dee0c0eb412fb08f498e38e90e86" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 39775 } health_report { overall_health: HEALTHY } } } I0408 18:33:42.345614 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 9307 W0408 18:33:42.349326 9098 connection.cc:513] server connection from 127.6.54.1:53943 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:42.349514 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 9087 W0408 18:33:42.352099 9282 connection.cc:513] client connection to 127.6.54.62:36737 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:42.352133 9086 connection.cc:513] client connection to 127.6.54.62:36737 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader/0 (1379 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader/1 I0408 18:33:42.354655 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:42.387449 9451 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:42.387604 9451 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:42.388311 9451 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:42.388324 9451 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:42.388327 9451 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:42.389326 9451 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.9451 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:42.390208 9451 master_main.cc:73] Initializing master server... I0408 18:33:42.390409 9451 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 30011us I0408 18:33:42.390605 9451 fs_manager.cc:260] Metadata directory not provided I0408 18:33:42.390640 9451 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:42.390713 9451 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:42.390730 9451 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:42.392038 9451 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "90b172b4dccc4129acf773a9d4dd24d5" format_stamp: "Formatted at 2019-04-08 18:33:42 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:42.392220 9451 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "90b172b4dccc4129acf773a9d4dd24d5" format_stamp: "Formatted at 2019-04-08 18:33:42 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:42.393501 9451 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0408 18:33:42.393757 9451 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:42.393790 9451 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:42.394389 9451 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:42.394428 9451 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/wal uuid: "90b172b4dccc4129acf773a9d4dd24d5" format_stamp: "Formatted at 2019-04-08 18:33:42 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:42.394474 9451 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:42.419991 9451 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:42.420069 9451 master_main.cc:76] Starting Master server... I0408 18:33:42.420204 9466 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:42.420225 9466 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:42.420228 9466 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:42.422196 9451 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:43405 I0408 18:33:42.422272 9451 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:42.422286 9451 webserver.cc:184] Document root disabled I0408 18:33:42.422490 9451 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:42875/ I0408 18:33:42.422719 9451 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:42.423249 9510 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:42.425292 9510 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5: Bootstrap starting. I0408 18:33:42.425701 9510 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5: No blocks or log segments found. Creating new log. I0408 18:33:42.425899 9510 log.cc:520] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5: Log is configured to *not* fsync() on all Append() calls I0408 18:33:42.426296 9510 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5: No bootstrap required, opened a new log I0408 18:33:42.427482 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 9451 I0408 18:33:42.427496 9510 raft_consensus.cc:304] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b172b4dccc4129acf773a9d4dd24d5" member_type: VOTER } I0408 18:33:42.427521 9510 raft_consensus.cc:330] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:42.427547 9510 raft_consensus.cc:605] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b172b4dccc4129acf773a9d4dd24d5, State: Initialized, Role: FOLLOWER I0408 18:33:42.427659 9510 consensus_queue.cc:226] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b172b4dccc4129acf773a9d4dd24d5" member_type: VOTER } I0408 18:33:42.427685 9510 raft_consensus.cc:344] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:42.427703 9510 raft_consensus.cc:436] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:42.427728 9510 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5: Snoozing failure detection for 2.013s (starting election) I0408 18:33:42.427747 9510 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:42.427762 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:43405 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_segment_size_mb=1 --log_compression_codec=NO_COMPRESSION with env {} I0408 18:33:42.428160 9510 raft_consensus.cc:458] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b172b4dccc4129acf773a9d4dd24d5" member_type: VOTER } I0408 18:33:42.428220 9510 leader_election.cc:258] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:42.428400 9512 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5: Snoozing failure detection for 1.728s (election complete) I0408 18:33:42.428427 9512 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:42.428503 9512 raft_consensus.cc:572] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [term 1 LEADER]: Becoming Leader. State: Replica: 90b172b4dccc4129acf773a9d4dd24d5, State: Running, Role: LEADER I0408 18:33:42.428606 9512 consensus_queue.cc:203] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b172b4dccc4129acf773a9d4dd24d5" member_type: VOTER } I0408 18:33:42.428844 9510 sys_catalog.cc:335] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "90b172b4dccc4129acf773a9d4dd24d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b172b4dccc4129acf773a9d4dd24d5" member_type: VOTER } } I0408 18:33:42.428884 9510 sys_catalog.cc:338] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [sys.catalog]: This master's current role is: LEADER I0408 18:33:42.428970 9512 sys_catalog.cc:335] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90b172b4dccc4129acf773a9d4dd24d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b172b4dccc4129acf773a9d4dd24d5" member_type: VOTER } } I0408 18:33:42.429033 9512 sys_catalog.cc:338] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [sys.catalog]: This master's current role is: LEADER I0408 18:33:42.429033 9510 sys_catalog.cc:422] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:42.428856 9513 sys_catalog.cc:335] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b172b4dccc4129acf773a9d4dd24d5. Latest consensus state: current_term: 1 leader_uuid: "90b172b4dccc4129acf773a9d4dd24d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b172b4dccc4129acf773a9d4dd24d5" member_type: VOTER } } I0408 18:33:42.429122 9513 sys_catalog.cc:338] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5 [sys.catalog]: This master's current role is: LEADER I0408 18:33:42.429369 9451 master_main.cc:79] Master server successfully started. W0408 18:33:42.429621 9518 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:42.429643 9518 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:42.429677 9517 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:42.429790 9517 catalog_manager.cc:983] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:42.460837 9516 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:42.460979 9516 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:42.461666 9516 flags.cc:412] Enabled experimental flag: --log_compression_codec=NO_COMPRESSION W0408 18:33:42.461686 9516 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:42.461704 9516 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:42.462635 9516 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=NO_COMPRESSION --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:43405 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.9516 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:42.463604 9516 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:42.463847 9516 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 30011us I0408 18:33:42.463986 9516 fs_manager.cc:260] Metadata directory not provided I0408 18:33:42.464018 9516 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:42.464089 9516 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:42.464107 9516 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:42.465338 9516 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "0934d20cbde545548385d7d462e3d37d" format_stamp: "Formatted at 2019-04-08 18:33:42 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:42.465512 9516 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "0934d20cbde545548385d7d462e3d37d" format_stamp: "Formatted at 2019-04-08 18:33:42 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:42.467470 9516 fs_manager.cc:495] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s I0408 18:33:42.467797 9516 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:42.467831 9516 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:42.468417 9516 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:42.468453 9516 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "0934d20cbde545548385d7d462e3d37d" format_stamp: "Formatted at 2019-04-08 18:33:42 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:42.468494 9516 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:42.471390 9517 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:42.472348 9517 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:42.476234 9517 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 90b172b4dccc4129acf773a9d4dd24d5: Generated new TSK 0 I0408 18:33:42.484622 9516 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:42.484819 9516 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:42.484889 9516 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:42.489061 9516 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:43603 I0408 18:33:42.489150 9516 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:42.489171 9516 webserver.cc:184] Document root disabled I0408 18:33:42.489423 9516 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36789/ I0408 18:33:42.489677 9516 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:42.490067 9516 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:42.490085 9641 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:42.490126 9641 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:42.490131 9641 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:42.491245 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 9516 I0408 18:33:42.495354 9640 heartbeater.cc:346] Connected to a master server at 127.6.54.62:43405 I0408 18:33:42.495402 9640 heartbeater.cc:426] Registering TS with master... I0408 18:33:42.495502 9640 heartbeater.cc:475] Master 127.6.54.62:43405 requested a full tablet report, sending... I0408 18:33:42.495986 9477 ts_manager.cc:81] Registered new tserver with Master: 0934d20cbde545548385d7d462e3d37d (127.6.54.1:43603) I0408 18:33:42.497010 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:42.497212 9477 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:43761 I0408 18:33:42.502172 6360 test_util.cc:200] Using random seed: 326121244 I0408 18:33:42.507381 9476 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44536: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:42.514101 9577 tablet_service.cc:758] Processing CreateTablet for tablet 87931ba9f69a4ce792f150022453d0b9 (table=test-workload [id=eb1a4be6af3d4924be6d5e8a61dc1abc]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:42.514250 9577 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 87931ba9f69a4ce792f150022453d0b9. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:42.516070 9577 ts_tablet_manager.cc:1080] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:42.519218 9658 ts_tablet_manager.cc:938] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Bootstrapping tablet I0408 18:33:42.519296 9658 tablet_bootstrap.cc:436] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Bootstrap starting. I0408 18:33:42.519713 9658 tablet_bootstrap.cc:581] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: No blocks or log segments found. Creating new log. W0408 18:33:42.519891 9658 compression_codec.cc:281] Unable to recognize the compression codec 'NO_COMPRESSION' using no compression as default. I0408 18:33:42.519994 9658 log.cc:520] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Log is configured to *not* fsync() on all Append() calls I0408 18:33:42.520370 9658 tablet_bootstrap.cc:436] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: No bootstrap required, opened a new log I0408 18:33:42.520421 9658 ts_tablet_manager.cc:955] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:42.521487 9658 raft_consensus.cc:304] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } } I0408 18:33:42.521550 9658 raft_consensus.cc:330] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:42.521574 9658 raft_consensus.cc:605] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0934d20cbde545548385d7d462e3d37d, State: Initialized, Role: FOLLOWER I0408 18:33:42.521715 9658 consensus_queue.cc:226] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } } I0408 18:33:42.521742 9658 raft_consensus.cc:344] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:42.521757 9658 raft_consensus.cc:436] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:42.521782 9658 raft_consensus.cc:2700] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Snoozing failure detection for 1.529s (starting election) I0408 18:33:42.521796 9658 raft_consensus.cc:2752] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:42.522238 9658 raft_consensus.cc:458] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } } I0408 18:33:42.522306 9658 leader_election.cc:258] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:42.522505 9658 ts_tablet_manager.cc:983] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:42.522512 9640 heartbeater.cc:467] Master 127.6.54.62:43405 was elected leader, sending a full tablet report... I0408 18:33:42.523222 9660 raft_consensus.cc:2700] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Snoozing failure detection for 1.595s (election complete) I0408 18:33:42.523249 9660 raft_consensus.cc:2507] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:42.523319 9660 raft_consensus.cc:572] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 1 LEADER]: Becoming Leader. State: Replica: 0934d20cbde545548385d7d462e3d37d, State: Running, Role: LEADER I0408 18:33:42.523386 9660 consensus_queue.cc:203] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } } I0408 18:33:42.523967 9476 catalog_manager.cc:3595] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d reported cstate change: term changed from 0 to 1, leader changed from to 0934d20cbde545548385d7d462e3d37d (127.6.54.1). New cstate: current_term: 1 leader_uuid: "0934d20cbde545548385d7d462e3d37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } health_report { overall_health: HEALTHY } } } W0408 18:33:42.528798 9634 generic_service.cc:126] {username='slave'} at 127.0.0.1:57612 forcing change of non-runtime-safe flag fault_crash_before_write_log_segment_header I0408 18:33:42.528846 9634 generic_service.cc:146] {username='slave'} at 127.0.0.1:57612 changed flags via RPC: fault_crash_before_write_log_segment_header from '0' to '0.9' I0408 18:33:42.579968 9663 log.cc:644] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Max segment size reached. Starting new segment allocation W0408 18:33:42.581105 9663 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:42.581126 9663 fault_injection.cc:42] THIS SERVER MAY CRASH! E0408 18:33:42.581133 9663 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_write_log_segment_header (process will exit) W0408 18:33:42.584643 9450 connection.cc:513] client connection to 127.6.54.1:43603 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:42.584681 9462 connection.cc:513] client connection to 127.6.54.1:43603 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:42.584738 9654 connection.cc:659] client connection to 127.6.54.1:43603 send error: Network error: failed to write to TLS socket: Connection reset by peer W0408 18:33:42.584681 9461 connection.cc:513] server connection from 127.6.54.1:43761 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:42.584758 9654 meta_cache.cc:219] Tablet 87931ba9f69a4ce792f150022453d0b9: Replica 0934d20cbde545548385d7d462e3d37d (127.6.54.1:43603) has failed: Network error: failed to write to TLS socket: Connection reset by peer W0408 18:33:42.584831 9654 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Success W0408 18:33:43.573899 9654 batcher.cc:349] Timed out: Failed to write batch of 50 ops to tablet 87931ba9f69a4ce792f150022453d0b9 after 41 attempt(s): Failed to write to server: (no server available): Write(tablet: 87931ba9f69a4ce792f150022453d0b9, num_ops: 50, num_attempts: 41) passed its deadline: Network error: failed to write to TLS socket: Connection reset by peer I0408 18:33:43.576649 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:43603 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=36789 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:43405 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_segment_size_mb=1 --log_compression_codec=NO_COMPRESSION with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:43.609236 9671 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:43.609383 9671 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:43.610080 9671 flags.cc:412] Enabled experimental flag: --log_compression_codec=NO_COMPRESSION W0408 18:33:43.610101 9671 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:43.610118 9671 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:43.611073 9671 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=NO_COMPRESSION --log_segment_size_mb=1 --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:43603 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=36789 --tserver_master_addrs=127.6.54.62:43405 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.9671 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:43.612030 9671 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:43.612278 9671 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 30511us I0408 18:33:43.612421 9671 fs_manager.cc:260] Metadata directory not provided I0408 18:33:43.612468 9671 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:43.613481 9671 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:43.613737 9671 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:43.613770 9671 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:43.614410 9671 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:43.614825 9671 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "0934d20cbde545548385d7d462e3d37d" format_stamp: "Formatted at 2019-04-08 18:33:42 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:43.614868 9671 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:43.633126 9671 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:43.633301 9671 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:43.633319 9671 ts_tablet_manager.cc:915] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Loading tablet metadata I0408 18:33:43.633868 9671 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:43.634331 9671 ts_tablet_manager.cc:1080] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:43.634436 9671 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:43.634444 9686 ts_tablet_manager.cc:938] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Bootstrapping tablet I0408 18:33:43.634541 9686 tablet_bootstrap.cc:436] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Bootstrap starting. W0408 18:33:43.635512 9686 log_util.cc:472] Log segment file /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/87931ba9f69a4ce792f150022453d0b9.recovery/wal-000000002 has 12 initial NULL bytes instead of magic and header length: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 and will be treated as a blank segment. W0408 18:33:43.635567 9686 log_reader.cc:149] Ignoring log segment wal-000000002 since it was uninitialized (probably left after a prior tablet server crash) W0408 18:33:43.636029 9686 compression_codec.cc:281] Unable to recognize the compression codec 'NO_COMPRESSION' using no compression as default. I0408 18:33:43.636129 9686 log.cc:520] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Log is configured to *not* fsync() on all Append() calls I0408 18:33:43.640105 9671 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:43603 I0408 18:33:43.640184 9671 webserver.cc:173] Starting webserver on 127.6.54.1:36789 I0408 18:33:43.640198 9671 webserver.cc:184] Document root disabled I0408 18:33:43.640426 9671 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36789/ I0408 18:33:43.640617 9671 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:43.640941 9671 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:43.641328 9793 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:43.641351 9793 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:43.641356 9793 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:43.642879 9686 log.cc:644] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Max segment size reached. Starting new segment allocation I0408 18:33:43.643406 9686 tablet_bootstrap.cc:436] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0408 18:33:43.643707 9686 tablet_bootstrap.cc:436] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Bootstrap complete. I0408 18:33:43.644026 9686 ts_tablet_manager.cc:955] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s I0408 18:33:43.645210 9792 heartbeater.cc:346] Connected to a master server at 127.6.54.62:43405 I0408 18:33:43.645256 9792 heartbeater.cc:426] Registering TS with master... I0408 18:33:43.645299 9686 raft_consensus.cc:304] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } } I0408 18:33:43.645350 9792 heartbeater.cc:475] Master 127.6.54.62:43405 requested a full tablet report, sending... I0408 18:33:43.645699 9686 raft_consensus.cc:605] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0934d20cbde545548385d7d462e3d37d, State: Initialized, Role: FOLLOWER I0408 18:33:43.645807 9686 consensus_queue.cc:226] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } } I0408 18:33:43.645838 9686 raft_consensus.cc:344] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:43.645908 9686 raft_consensus.cc:436] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:43.645951 9686 raft_consensus.cc:2700] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Snoozing failure detection for 1.941s (starting election) I0408 18:33:43.645963 9686 raft_consensus.cc:2752] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:43.646265 9477 ts_manager.cc:87] Re-registered known tserver with Master: 0934d20cbde545548385d7d462e3d37d (127.6.54.1:43603) I0408 18:33:43.646632 9686 raft_consensus.cc:458] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } } I0408 18:33:43.646688 9686 leader_election.cc:258] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:43.646836 9477 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:44061 I0408 18:33:43.646875 9686 ts_tablet_manager.cc:983] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s I0408 18:33:43.646944 9797 raft_consensus.cc:2700] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Snoozing failure detection for 1.688s (election complete) I0408 18:33:43.646970 9797 raft_consensus.cc:2507] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:43.647042 9797 raft_consensus.cc:572] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [term 2 LEADER]: Becoming Leader. State: Replica: 0934d20cbde545548385d7d462e3d37d, State: Running, Role: LEADER I0408 18:33:43.647117 9797 consensus_queue.cc:203] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } } I0408 18:33:43.647516 9800 log.cc:571] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d: Rolled over to a new log segment at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/87931ba9f69a4ce792f150022453d0b9/wal-000000002 I0408 18:33:43.647574 9792 heartbeater.cc:467] Master 127.6.54.62:43405 was elected leader, sending a full tablet report... I0408 18:33:43.647915 9477 catalog_manager.cc:3595] T 87931ba9f69a4ce792f150022453d0b9 P 0934d20cbde545548385d7d462e3d37d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "0934d20cbde545548385d7d462e3d37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0934d20cbde545548385d7d462e3d37d" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43603 } health_report { overall_health: HEALTHY } } } I0408 18:33:43.649924 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 9671 I0408 18:33:43.657744 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 9671 W0408 18:33:43.660903 9462 connection.cc:513] server connection from 127.6.54.1:44061 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:43.661031 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 9451 W0408 18:33:43.663549 9646 connection.cc:513] client connection to 127.6.54.62:43405 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:43.663589 9450 connection.cc:513] client connection to 127.6.54.62:43405 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestCrashBeforeWriteLogSegmentHeader/1 (1311 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestChangeMaxCellSize/0 I0408 18:33:43.665984 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data --block_manager=file --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_tombstone_evicted_tablet_replicas=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:43.699028 9815 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:43.699203 9815 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:43.699942 9815 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:43.699959 9815 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:43.699965 9815 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:43.701036 9815 master_main.cc:66] Master server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/wal --master_tombstone_evicted_tablet_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.9815 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:43.701939 9815 master_main.cc:73] Initializing master server... I0408 18:33:43.702158 9815 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 30511us I0408 18:33:43.702342 9815 fs_manager.cc:260] Metadata directory not provided I0408 18:33:43.702378 9815 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:43.702456 9815 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:43.702476 9815 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:43.703815 9815 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" format_stamp: "Formatted at 2019-04-08 18:33:43 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:43.703991 9815 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" format_stamp: "Formatted at 2019-04-08 18:33:43 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:43.705175 9815 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:43.705402 9815 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:43.705436 9815 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:43.705541 9815 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:43.705580 9815 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/wal uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" format_stamp: "Formatted at 2019-04-08 18:33:43 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:43.705621 9815 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:43.743758 9815 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:43.743834 9815 master_main.cc:76] Starting Master server... I0408 18:33:43.743978 9830 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:43.744000 9830 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:43.744006 9830 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:43.746085 9815 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:35969 I0408 18:33:43.746170 9815 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:43.746184 9815 webserver.cc:184] Document root disabled I0408 18:33:43.746400 9815 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:43583/ I0408 18:33:43.746655 9815 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:43.747102 9874 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:43.749215 9874 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb: Bootstrap starting. I0408 18:33:43.749688 9874 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb: No blocks or log segments found. Creating new log. I0408 18:33:43.749903 9874 log.cc:520] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb: Log is configured to *not* fsync() on all Append() calls I0408 18:33:43.750280 9874 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb: No bootstrap required, opened a new log I0408 18:33:43.751324 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 9815 I0408 18:33:43.751569 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:35969 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} I0408 18:33:43.751652 9874 raft_consensus.cc:304] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" member_type: VOTER } I0408 18:33:43.751696 9874 raft_consensus.cc:330] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:43.751718 9874 raft_consensus.cc:605] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b3684dbaf64f27aab15df9d5ee83fb, State: Initialized, Role: FOLLOWER I0408 18:33:43.751844 9874 consensus_queue.cc:226] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" member_type: VOTER } I0408 18:33:43.751868 9874 raft_consensus.cc:344] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:43.751883 9874 raft_consensus.cc:436] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:43.751911 9874 raft_consensus.cc:2700] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb: Snoozing failure detection for 1.738s (starting election) I0408 18:33:43.751929 9874 raft_consensus.cc:2752] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:43.752342 9874 raft_consensus.cc:458] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" member_type: VOTER } I0408 18:33:43.752411 9874 leader_election.cc:258] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:43.752609 9874 sys_catalog.cc:335] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" member_type: VOTER } } I0408 18:33:43.752636 9874 sys_catalog.cc:338] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [sys.catalog]: This master's current role is: FOLLOWER I0408 18:33:43.752738 9874 sys_catalog.cc:422] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:43.753005 9815 master_main.cc:79] Master server successfully started. I0408 18:33:43.752889 9876 raft_consensus.cc:2700] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb: Snoozing failure detection for 1.855s (election complete) I0408 18:33:43.753046 9876 raft_consensus.cc:2507] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:43.753134 9876 raft_consensus.cc:572] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [term 1 LEADER]: Becoming Leader. State: Replica: f8b3684dbaf64f27aab15df9d5ee83fb, State: Running, Role: LEADER I0408 18:33:43.753206 9876 consensus_queue.cc:203] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" member_type: VOTER } I0408 18:33:43.753397 9876 sys_catalog.cc:335] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b3684dbaf64f27aab15df9d5ee83fb. Latest consensus state: current_term: 1 leader_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" member_type: VOTER } } I0408 18:33:43.753867 9876 sys_catalog.cc:338] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [sys.catalog]: This master's current role is: LEADER W0408 18:33:43.753783 9879 catalog_manager.cc:1018] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:43.755245 9879 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:43.753471 9877 sys_catalog.cc:335] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b3684dbaf64f27aab15df9d5ee83fb" member_type: VOTER } } I0408 18:33:43.755276 9877 sys_catalog.cc:338] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb [sys.catalog]: This master's current role is: LEADER I0408 18:33:43.755318 9883 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:43.755434 9883 catalog_manager.cc:983] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:43.785063 9878 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:43.785218 9878 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:43.785923 9878 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:43.785949 9878 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:43.786924 9878 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:35969 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.9878 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:43.787964 9878 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:43.788206 9878 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 30511us I0408 18:33:43.788352 9878 fs_manager.cc:260] Metadata directory not provided I0408 18:33:43.788380 9878 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:43.788455 9878 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:43.788475 9878 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:43.789760 9878 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" format_stamp: "Formatted at 2019-04-08 18:33:43 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:43.789937 9878 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" format_stamp: "Formatted at 2019-04-08 18:33:43 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:43.790927 9883 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:43.791432 9878 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:43.791757 9878 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:43.791791 9878 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:43.791891 9878 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0408 18:33:43.791927 9878 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" format_stamp: "Formatted at 2019-04-08 18:33:43 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:43.791991 9883 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:43.792017 9878 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:43.799443 9883 catalog_manager.cc:3871] T 00000000000000000000000000000000 P f8b3684dbaf64f27aab15df9d5ee83fb: Generated new TSK 0 I0408 18:33:43.822104 9878 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:43.822335 9878 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:43.822407 9878 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:43.826838 9878 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:45017 I0408 18:33:43.826943 9878 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:43.826959 9878 webserver.cc:184] Document root disabled I0408 18:33:43.827250 9878 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:38915/ I0408 18:33:43.827486 9878 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:43.827771 9878 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:43.831459 10005 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:43.831480 10005 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:43.831483 10005 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:43.835047 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 9878 I0408 18:33:43.840147 10004 heartbeater.cc:346] Connected to a master server at 127.6.54.62:35969 I0408 18:33:43.840193 10004 heartbeater.cc:426] Registering TS with master... I0408 18:33:43.840298 10004 heartbeater.cc:475] Master 127.6.54.62:35969 requested a full tablet report, sending... I0408 18:33:43.840736 9841 ts_manager.cc:81] Registered new tserver with Master: fd5ebd1de3774e49b02d250ef2fcffe7 (127.6.54.1:45017) I0408 18:33:43.841563 9841 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:38185 I0408 18:33:43.841874 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:43.846475 6360 test_util.cc:200] Using random seed: 327465547 I0408 18:33:43.851459 9841 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50556: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:43.859946 9939 tablet_service.cc:758] Processing CreateTablet for tablet b9cd2372808148f0a622c27c32ac974e (table=test-workload [id=b10e5561b1784357b03596d5960438aa]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:43.860146 9939 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet b9cd2372808148f0a622c27c32ac974e. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:43.862934 9939 ts_tablet_manager.cc:1080] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:43.863211 10022 ts_tablet_manager.cc:938] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Bootstrapping tablet I0408 18:33:43.863293 10022 tablet_bootstrap.cc:436] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Bootstrap starting. I0408 18:33:43.863732 10022 tablet_bootstrap.cc:581] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: No blocks or log segments found. Creating new log. I0408 18:33:43.863943 10022 log.cc:520] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Log is configured to *not* fsync() on all Append() calls I0408 18:33:43.867522 10022 tablet_bootstrap.cc:436] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: No bootstrap required, opened a new log I0408 18:33:43.867596 10022 ts_tablet_manager.cc:955] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I0408 18:33:43.868676 10022 raft_consensus.cc:304] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } } I0408 18:33:43.868719 10022 raft_consensus.cc:330] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:43.868741 10022 raft_consensus.cc:605] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5ebd1de3774e49b02d250ef2fcffe7, State: Initialized, Role: FOLLOWER I0408 18:33:43.868854 10022 consensus_queue.cc:226] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } } I0408 18:33:43.868877 10022 raft_consensus.cc:344] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:43.868892 10022 raft_consensus.cc:436] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:43.868926 10022 raft_consensus.cc:2700] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Snoozing failure detection for 1.608s (starting election) I0408 18:33:43.868943 10022 raft_consensus.cc:2752] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:43.869379 10022 raft_consensus.cc:458] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } } I0408 18:33:43.869437 10022 leader_election.cc:258] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:43.869645 10022 ts_tablet_manager.cc:983] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:43.869655 10004 heartbeater.cc:467] Master 127.6.54.62:35969 was elected leader, sending a full tablet report... I0408 18:33:43.871217 10024 raft_consensus.cc:2700] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Snoozing failure detection for 1.537s (election complete) I0408 18:33:43.871242 10024 raft_consensus.cc:2507] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:43.871309 10024 raft_consensus.cc:572] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 LEADER]: Becoming Leader. State: Replica: fd5ebd1de3774e49b02d250ef2fcffe7, State: Running, Role: LEADER I0408 18:33:43.871367 10024 consensus_queue.cc:203] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } } I0408 18:33:43.872076 9841 catalog_manager.cc:3595] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 reported cstate change: term changed from 0 to 1, leader changed from to fd5ebd1de3774e49b02d250ef2fcffe7 (127.6.54.1). New cstate: current_term: 1 leader_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } health_report { overall_health: HEALTHY } } } I0408 18:33:43.968430 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 9878 W0408 18:33:43.971513 10017 connection.cc:513] client connection to 127.6.54.1:45017 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:43.971586 9825 connection.cc:513] server connection from 127.6.54.1:38185 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:43.971833 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:45017 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=38915 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:35969 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --max_cell_size_bytes=1000 with env {} W0408 18:33:43.971904 9825 connection.cc:513] client connection to 127.6.54.1:45017 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:44.004421 10035 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:44.004575 10035 flags.cc:412] Enabled unsafe flag: --max_cell_size_bytes=1000 W0408 18:33:44.004587 10035 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:44.005285 10035 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:44.005311 10035 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:44.006295 10035 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:45017 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=38915 --max_cell_size_bytes=1000 --tserver_master_addrs=127.6.54.62:35969 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10035 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:44.007344 10035 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:44.007611 10035 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31011us I0408 18:33:44.007757 10035 fs_manager.cc:260] Metadata directory not provided I0408 18:33:44.007797 10035 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:44.008795 10035 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:44.008992 10035 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:44.009018 10035 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:44.009109 10035 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:44.009483 10035 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" format_stamp: "Formatted at 2019-04-08 18:33:43 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.009533 10035 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:44.024329 10035 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:44.024549 10035 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:44.024569 10035 ts_tablet_manager.cc:915] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Loading tablet metadata I0408 18:33:44.025115 10035 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:44.025645 10035 ts_tablet_manager.cc:1080] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:44.025776 10035 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:44.025794 10050 ts_tablet_manager.cc:938] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Bootstrapping tablet I0408 18:33:44.025866 10050 tablet_bootstrap.cc:436] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Bootstrap starting. I0408 18:33:44.035035 10035 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:45017 I0408 18:33:44.035127 10035 webserver.cc:173] Starting webserver on 127.6.54.1:38915 I0408 18:33:44.035181 10035 webserver.cc:184] Document root disabled I0408 18:33:44.035284 10050 log.cc:520] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Log is configured to *not* fsync() on all Append() calls I0408 18:33:44.035461 10035 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:38915/ I0408 18:33:44.035712 10035 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:44.036026 10035 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:44.036258 10157 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:44.036280 10157 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:44.036286 10157 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:44.040165 10156 heartbeater.cc:346] Connected to a master server at 127.6.54.62:35969 I0408 18:33:44.040211 10156 heartbeater.cc:426] Registering TS with master... I0408 18:33:44.040297 10156 heartbeater.cc:475] Master 127.6.54.62:35969 requested a full tablet report, sending... I0408 18:33:44.040740 9841 ts_manager.cc:87] Re-registered known tserver with Master: fd5ebd1de3774e49b02d250ef2fcffe7 (127.6.54.1:45017) I0408 18:33:44.041221 9841 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:42327 I0408 18:33:44.041822 10050 tablet_bootstrap.cc:1138] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: ReplayState: Previous OpId: 1.2, Committed OpId: 1.2, Pending Replicates: 0, Pending Commits: 0 E0408 18:33:44.042464 10050 ts_tablet_manager.cc:976] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Tablet failed to bootstrap: Corruption: Failed log replay. Reason: Debug Info: Error playing entry 4 of segment 1 of tablet b9cd2372808148f0a622c27c32ac974e. Segment path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/b9cd2372808148f0a622c27c32ac974e.recovery/wal-000000001. Entry: type: COMMIT commit { op_type: WRITE_OP commited_op_id { term: 1 index: 2 } result { ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mu...: Failed to play WRITE_OP request. ReplicateMsg: { id { term: 1 index: 2 } timestamp: 6368249544325132288 op_type: WRITE_OP write_request { tablet_id: "b9cd2372808148f0a622c27c32ac974e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\357\233\342ni\333\010\023\000\000\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\345\245\374B\347B\324Y\020\'\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\2278\342Wm\264-; N\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\000\312YK\020\2004\2550u\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000dM=w6\276\205%@\234\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\021\020\261)<\343\005\341P\303\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\261\032\301H\265\241A\001`\352\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000TZ\201<~\200j\016p\021\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\326C\375[\272\366\310\033\2008\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\301\032\024XX\321\210m\220_\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\342\033\265\026}\030\271\332\240\206\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\354\225\2503\026vy\274\260\255\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000]\260\022\nzK\236\366\300\324\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\026\250\003#\365G\205\261\320\373\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000x\003\0368\320\335\006\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000Bj\275Q\273wP[\340\004\007\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000I\261\344+p|Ak\360+\007\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\322\263\311\021\225\330\202\345\000S\007\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000Xo\336SC\3568x\020z\007\000\000\000\000\000\020\'\000\000\000\000\000\000" indirect_data: "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 I0408 18:33:44.043742 10050 ts_tablet_manager.cc:955] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.010s I0408 18:33:44.043896 10050 tablet_replica.cc:262] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: stopping tablet replica I0408 18:33:44.044052 10050 raft_consensus.cc:2012] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Raft consensus shutting down. I0408 18:33:44.044095 10050 raft_consensus.cc:2039] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Raft consensus is shut down! I0408 18:33:44.045403 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 10035 I0408 18:33:44.070691 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 10035 W0408 18:33:44.073899 9824 connection.cc:513] server connection from 127.6.54.1:42327 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:44.073907 9814 connection.cc:513] client connection to 127.6.54.1:45017 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:44.074101 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=file --rpc_bind_addresses=127.6.54.1:45017 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=38915 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:35969 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:44.108306 10160 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:44.108444 10160 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:44.109134 10160 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:44.109154 10160 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:44.110133 10160 tablet_server_main.cc:72] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:45017 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=38915 --tserver_master_addrs=127.6.54.62:35969 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10160 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:44.111181 10160 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:44.111400 10160 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31011us I0408 18:33:44.111534 10160 fs_manager.cc:260] Metadata directory not provided I0408 18:33:44.111562 10160 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:44.112506 10160 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:44.112727 10160 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:44.112751 10160 file_cache.cc:470] Constructed file cache fbm with capacity 419430 I0408 18:33:44.112823 10160 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:44.113171 10160 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" format_stamp: "Formatted at 2019-04-08 18:33:43 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.113209 10160 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0408 18:33:44.138046 10160 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:44.138262 10160 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:44.138283 10160 ts_tablet_manager.cc:915] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Loading tablet metadata I0408 18:33:44.138865 10160 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:44.139394 10160 ts_tablet_manager.cc:1080] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:44.139519 10175 ts_tablet_manager.cc:938] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Bootstrapping tablet I0408 18:33:44.139520 10160 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:44.139629 10175 tablet_bootstrap.cc:436] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Bootstrap starting. I0408 18:33:44.140056 10175 tablet_bootstrap.cc:645] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Previous recovery directory found at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/b9cd2372808148f0a622c27c32ac974e.recovery: Replaying log files from this location instead of /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/b9cd2372808148f0a622c27c32ac974e I0408 18:33:44.140095 10175 tablet_bootstrap.cc:652] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Deleting old log files from previous recovery attempt in /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/wal/wals/b9cd2372808148f0a622c27c32ac974e I0408 18:33:44.144668 10160 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:45017 I0408 18:33:44.147403 10160 webserver.cc:173] Starting webserver on 127.6.54.1:38915 I0408 18:33:44.147424 10160 webserver.cc:184] Document root disabled I0408 18:33:44.147703 10160 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:38915/ I0408 18:33:44.147745 10175 log.cc:520] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Log is configured to *not* fsync() on all Append() calls I0408 18:33:44.147975 10160 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:44.148334 10160 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:44.148700 10282 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:44.148725 10282 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:44.148730 10282 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:44.152034 10281 heartbeater.cc:346] Connected to a master server at 127.6.54.62:35969 I0408 18:33:44.152074 10281 heartbeater.cc:426] Registering TS with master... I0408 18:33:44.152160 10281 heartbeater.cc:475] Master 127.6.54.62:35969 requested a full tablet report, sending... I0408 18:33:44.152624 9841 ts_manager.cc:87] Re-registered known tserver with Master: fd5ebd1de3774e49b02d250ef2fcffe7 (127.6.54.1:45017) I0408 18:33:44.153102 9841 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:57257 I0408 18:33:44.153681 10175 tablet_bootstrap.cc:436] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0408 18:33:44.153915 10175 tablet_bootstrap.cc:436] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Bootstrap complete. I0408 18:33:44.154551 10175 ts_tablet_manager.cc:955] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s I0408 18:33:44.155565 10175 raft_consensus.cc:304] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } } I0408 18:33:44.155917 10175 raft_consensus.cc:605] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5ebd1de3774e49b02d250ef2fcffe7, State: Initialized, Role: FOLLOWER I0408 18:33:44.156044 10175 consensus_queue.cc:226] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } } I0408 18:33:44.156066 10175 raft_consensus.cc:344] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:44.156085 10175 raft_consensus.cc:436] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:44.156118 10175 raft_consensus.cc:2700] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Snoozing failure detection for 1.710s (starting election) I0408 18:33:44.156149 10175 raft_consensus.cc:2752] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:44.156749 10175 raft_consensus.cc:458] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } } I0408 18:33:44.156795 10175 leader_election.cc:258] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:44.156980 10285 raft_consensus.cc:2700] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Snoozing failure detection for 2.195s (election complete) I0408 18:33:44.157001 10285 raft_consensus.cc:2507] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:44.157049 10281 heartbeater.cc:467] Master 127.6.54.62:35969 was elected leader, sending a full tablet report... I0408 18:33:44.157074 10285 raft_consensus.cc:572] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [term 2 LEADER]: Becoming Leader. State: Replica: fd5ebd1de3774e49b02d250ef2fcffe7, State: Running, Role: LEADER I0408 18:33:44.157061 10175 ts_tablet_manager.cc:983] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s I0408 18:33:44.157152 10285 consensus_queue.cc:203] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } } I0408 18:33:44.157438 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 10160 I0408 18:33:44.157673 9841 catalog_manager.cc:3595] T b9cd2372808148f0a622c27c32ac974e P fd5ebd1de3774e49b02d250ef2fcffe7 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5ebd1de3774e49b02d250ef2fcffe7" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 45017 } health_report { overall_health: HEALTHY } } } I0408 18:33:44.165706 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 10160 W0408 18:33:44.169582 9823 connection.cc:513] server connection from 127.6.54.1:57257 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:44.169714 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 9815 W0408 18:33:44.172307 10011 connection.cc:513] client connection to 127.6.54.62:35969 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:44.172332 9814 connection.cc:513] client connection to 127.6.54.62:35969 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestChangeMaxCellSize/0 (509 ms) [ RUN ] BlockManagerType/TsRecoveryITest.TestChangeMaxCellSize/1 I0408 18:33:44.174811 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_tombstone_evicted_tablet_replicas=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:44.207806 10299 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:44.207952 10299 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:44.208688 10299 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:44.208700 10299 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:44.208704 10299 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:44.209735 10299 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/wal --master_tombstone_evicted_tablet_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.10299 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:44.210608 10299 master_main.cc:73] Initializing master server... I0408 18:33:44.210813 10299 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31011us I0408 18:33:44.210994 10299 fs_manager.cc:260] Metadata directory not provided I0408 18:33:44.211028 10299 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:44.211107 10299 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:44.211124 10299 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:44.212404 10299 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.212597 10299 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.213806 10299 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0408 18:33:44.214062 10299 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:44.214098 10299 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:44.214699 10299 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:44.214732 10299 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/wal uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.214781 10299 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:44.276175 10299 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:44.276247 10299 master_main.cc:76] Starting Master server... I0408 18:33:44.276403 10314 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:44.276425 10314 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:44.276429 10314 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:44.278388 10299 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:44093 I0408 18:33:44.278461 10299 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:44.278478 10299 webserver.cc:184] Document root disabled I0408 18:33:44.278705 10299 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:42317/ I0408 18:33:44.278923 10299 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:44.279399 10358 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:44.281466 10358 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338: Bootstrap starting. I0408 18:33:44.281888 10358 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338: No blocks or log segments found. Creating new log. I0408 18:33:44.282096 10358 log.cc:520] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338: Log is configured to *not* fsync() on all Append() calls I0408 18:33:44.282459 10358 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338: No bootstrap required, opened a new log I0408 18:33:44.283696 10358 raft_consensus.cc:304] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" member_type: VOTER } I0408 18:33:44.283737 10358 raft_consensus.cc:330] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:44.283759 10358 raft_consensus.cc:605] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306a5e7e877747ebbe9bd6ea0e5d9338, State: Initialized, Role: FOLLOWER I0408 18:33:44.283867 10358 consensus_queue.cc:226] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" member_type: VOTER } I0408 18:33:44.283891 10358 raft_consensus.cc:344] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:44.283906 10358 raft_consensus.cc:436] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:44.283933 10358 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338: Snoozing failure detection for 2.164s (starting election) I0408 18:33:44.283951 10358 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:44.284364 10358 raft_consensus.cc:458] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" member_type: VOTER } I0408 18:33:44.284425 10358 leader_election.cc:258] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:44.284579 10360 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338: Snoozing failure detection for 1.746s (election complete) I0408 18:33:44.284611 10360 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:44.284679 10360 raft_consensus.cc:572] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [term 1 LEADER]: Becoming Leader. State: Replica: 306a5e7e877747ebbe9bd6ea0e5d9338, State: Running, Role: LEADER I0408 18:33:44.284754 10360 consensus_queue.cc:203] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" member_type: VOTER } I0408 18:33:44.284938 10360 sys_catalog.cc:335] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306a5e7e877747ebbe9bd6ea0e5d9338. Latest consensus state: current_term: 1 leader_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" member_type: VOTER } } I0408 18:33:44.284948 10358 sys_catalog.cc:335] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" member_type: VOTER } } I0408 18:33:44.284974 10360 sys_catalog.cc:338] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [sys.catalog]: This master's current role is: LEADER I0408 18:33:44.284986 10361 sys_catalog.cc:335] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a5e7e877747ebbe9bd6ea0e5d9338" member_type: VOTER } } I0408 18:33:44.285034 10361 sys_catalog.cc:338] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [sys.catalog]: This master's current role is: LEADER I0408 18:33:44.284976 10358 sys_catalog.cc:338] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [sys.catalog]: This master's current role is: LEADER I0408 18:33:44.285182 10358 sys_catalog.cc:422] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:44.285409 10299 master_main.cc:79] Master server successfully started. W0408 18:33:44.285607 10365 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:44.285625 10365 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:44.285671 10364 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:44.285787 10364 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0408 18:33:44.288188 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 10299 I0408 18:33:44.288422 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:44093 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} I0408 18:33:44.301983 10364 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:44.302923 10364 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:44.308894 10364 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 306a5e7e877747ebbe9bd6ea0e5d9338: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:44.323398 10367 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:44.323549 10367 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:44.324223 10367 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:44.324250 10367 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:44.325170 10367 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:44093 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10367 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:44.326115 10367 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:44.326347 10367 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31011us I0408 18:33:44.326491 10367 fs_manager.cc:260] Metadata directory not provided I0408 18:33:44.326534 10367 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:44.326606 10367 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:44.326625 10367 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:44.327903 10367 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.328088 10367 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.331421 10367 fs_manager.cc:495] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s I0408 18:33:44.331693 10367 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:44.331727 10367 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:44.332307 10367 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:44.332343 10367 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.332398 10367 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:44.351048 10367 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:44.351367 10367 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:44.351445 10367 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:44.355057 10367 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:44013 I0408 18:33:44.355139 10367 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:44.355176 10367 webserver.cc:184] Document root disabled I0408 18:33:44.355379 10367 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36947/ I0408 18:33:44.355595 10367 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:44.355950 10367 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:44.356036 10489 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:44.356048 10489 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:44.356055 10489 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:44.360572 10488 heartbeater.cc:346] Connected to a master server at 127.6.54.62:44093 I0408 18:33:44.360618 10488 heartbeater.cc:426] Registering TS with master... I0408 18:33:44.360707 10488 heartbeater.cc:475] Master 127.6.54.62:44093 requested a full tablet report, sending... I0408 18:33:44.361181 10325 ts_manager.cc:81] Registered new tserver with Master: a8420c2e1fef4bb6b00a09f1d95dd0ec (127.6.54.1:44013) I0408 18:33:44.362414 10325 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:33567 I0408 18:33:44.363503 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 10367 I0408 18:33:44.366238 6360 external_mini_cluster.cc:499] 1 TS(s) registered with all masters I0408 18:33:44.370354 6360 test_util.cc:200] Using random seed: 327989426 I0408 18:33:44.375414 10325 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47076: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0408 18:33:44.380656 10425 tablet_service.cc:758] Processing CreateTablet for tablet e8251b7be8b2495cb7a14bb6f3d6559c (table=test-workload [id=2fb8cdf402344c4e9248499afff8425b]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:44.380839 10425 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet e8251b7be8b2495cb7a14bb6f3d6559c. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:44.383339 10425 ts_tablet_manager.cc:1080] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:44.383486 10505 ts_tablet_manager.cc:938] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Bootstrapping tablet I0408 18:33:44.383631 10505 tablet_bootstrap.cc:436] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Bootstrap starting. I0408 18:33:44.384058 10505 tablet_bootstrap.cc:581] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: No blocks or log segments found. Creating new log. I0408 18:33:44.384292 10505 log.cc:520] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Log is configured to *not* fsync() on all Append() calls I0408 18:33:44.384714 10505 tablet_bootstrap.cc:436] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: No bootstrap required, opened a new log I0408 18:33:44.384776 10505 ts_tablet_manager.cc:955] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:44.386217 10505 raft_consensus.cc:304] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } } I0408 18:33:44.386265 10505 raft_consensus.cc:330] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:44.386284 10505 raft_consensus.cc:605] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8420c2e1fef4bb6b00a09f1d95dd0ec, State: Initialized, Role: FOLLOWER I0408 18:33:44.386405 10505 consensus_queue.cc:226] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } } I0408 18:33:44.386430 10505 raft_consensus.cc:344] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:44.386443 10505 raft_consensus.cc:436] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:44.386467 10505 raft_consensus.cc:2700] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Snoozing failure detection for 2.226s (starting election) I0408 18:33:44.386484 10505 raft_consensus.cc:2752] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:44.387064 10505 raft_consensus.cc:458] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } } I0408 18:33:44.387135 10505 leader_election.cc:258] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:44.387298 10507 raft_consensus.cc:2700] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Snoozing failure detection for 2.248s (election complete) I0408 18:33:44.387323 10507 raft_consensus.cc:2507] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:44.387359 10488 heartbeater.cc:467] Master 127.6.54.62:44093 was elected leader, sending a full tablet report... I0408 18:33:44.387399 10507 raft_consensus.cc:572] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 LEADER]: Becoming Leader. State: Replica: a8420c2e1fef4bb6b00a09f1d95dd0ec, State: Running, Role: LEADER I0408 18:33:44.387459 10507 consensus_queue.cc:203] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } } I0408 18:33:44.387554 10505 ts_tablet_manager.cc:983] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I0408 18:33:44.388213 10325 catalog_manager.cc:3595] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec reported cstate change: term changed from 0 to 1, leader changed from to a8420c2e1fef4bb6b00a09f1d95dd0ec (127.6.54.1). New cstate: current_term: 1 leader_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } health_report { overall_health: HEALTHY } } } I0408 18:33:44.472721 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 10367 W0408 18:33:44.475915 10501 connection.cc:513] client connection to 127.6.54.1:44013 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:44.475973 10309 connection.cc:513] server connection from 127.6.54.1:33567 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:44.475982 10310 connection.cc:513] client connection to 127.6.54.1:44013 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:44.476233 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:44013 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=36947 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:44093 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --max_cell_size_bytes=1000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:44.509196 10518 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:44.509352 10518 flags.cc:412] Enabled unsafe flag: --max_cell_size_bytes=1000 W0408 18:33:44.509358 10518 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:44.510053 10518 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:44.510084 10518 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:44.511060 10518 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:44013 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=36947 --max_cell_size_bytes=1000 --tserver_master_addrs=127.6.54.62:44093 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10518 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:44.512107 10518 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:44.512360 10518 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31011us I0408 18:33:44.512512 10518 fs_manager.cc:260] Metadata directory not provided I0408 18:33:44.512563 10518 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:44.513592 10518 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s I0408 18:33:44.513797 10518 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:44.513824 10518 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:44.514425 10518 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:44.514814 10518 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.514858 10518 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:44.536972 10518 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:44.537199 10518 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:44.537220 10518 ts_tablet_manager.cc:915] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Loading tablet metadata I0408 18:33:44.537829 10518 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:44.538345 10518 ts_tablet_manager.cc:1080] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:44.538511 10533 ts_tablet_manager.cc:938] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Bootstrapping tablet I0408 18:33:44.538509 10518 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:44.538641 10533 tablet_bootstrap.cc:436] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Bootstrap starting. I0408 18:33:44.543576 10518 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:44013 I0408 18:33:44.543675 10518 webserver.cc:173] Starting webserver on 127.6.54.1:36947 I0408 18:33:44.543692 10518 webserver.cc:184] Document root disabled I0408 18:33:44.543923 10518 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36947/ I0408 18:33:44.544157 10518 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:44.544518 10518 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:44.544555 10639 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:44.544649 10639 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:44.544663 10639 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:44.548382 10638 heartbeater.cc:346] Connected to a master server at 127.6.54.62:44093 I0408 18:33:44.548431 10638 heartbeater.cc:426] Registering TS with master... I0408 18:33:44.548512 10638 heartbeater.cc:475] Master 127.6.54.62:44093 requested a full tablet report, sending... I0408 18:33:44.548969 10321 ts_manager.cc:87] Re-registered known tserver with Master: a8420c2e1fef4bb6b00a09f1d95dd0ec (127.6.54.1:44013) I0408 18:33:44.549154 10533 log.cc:520] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Log is configured to *not* fsync() on all Append() calls I0408 18:33:44.549448 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 10518 I0408 18:33:44.549715 10321 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:38323 I0408 18:33:44.555470 10533 tablet_bootstrap.cc:1138] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: ReplayState: Previous OpId: 1.2, Committed OpId: 1.2, Pending Replicates: 0, Pending Commits: 0 E0408 18:33:44.556361 10533 ts_tablet_manager.cc:976] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Tablet failed to bootstrap: Corruption: Failed log replay. Reason: Debug Info: Error playing entry 4 of segment 1 of tablet e8251b7be8b2495cb7a14bb6f3d6559c. Segment path: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/e8251b7be8b2495cb7a14bb6f3d6559c.recovery/wal-000000001. Entry: type: COMMIT commit { op_type: WRITE_OP commited_op_id { term: 1 index: 2 } result { ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mutated_stores { mrs_id: 0 } } ops { mu...: Failed to play WRITE_OP request. ReplicateMsg: { id { term: 1 index: 2 } timestamp: 6368249546405625856 op_type: WRITE_OP write_request { tablet_id: "e8251b7be8b2495cb7a14bb6f3d6559c" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000$\270\261{\375\276c\364\000\000\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\375D\033\\\315\324\350\352\020\'\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\034$\213\005;\273h\203 N\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\327!\231\0373\230\265\2560u\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\340\361r/!<\200m@\234\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\371\325_#\253\010\200\366P\303\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\356\265\277o\305/Y\246`\352\000\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\030|\252\003\322\264\'\320p\021\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\0005\207\352N3[(:\2008\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000f\236|b\353J[\321\220_\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\344R\025\r\255@+\262\240\206\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000KN\243m\347\233\227g\260\255\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\243\216\242n\377\261(\366\300\324\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000$\277\371Y\302\347X]\320\373\001\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\277\323\202\036\r\010\320\322\340\"\002\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\0009\033^W\240\207\346n\360I\002\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\232\306\224MBL\376l\000q\002\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\255\272\303_\207\255\304\350\020\230\002\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\374j\355\030\0045\256\363 \277\002\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\273B\3029\302#\352\3420\346\002\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\364\244J/\345\014\325\230@\r\003\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000L\023xA\333\367\215@P4\003\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\310\350\224\003\374\017\2253`[\003\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000y{\326Z\302~\374np\202\003\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\241K\030aY7]>\200\251\003\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\261x)+\355$\006\001\220\320\003\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\311\324lo\025\363W\214\240\367\003\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000*\370\005ERh\010\361\260\036\004\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000T\366\301d\214\316\340\203\300E\004\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000p\346\253f\352\242*\311\320l\004\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\230\022\341\037dnZ\220\340\223\004\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000}\357\257,T\217Av\360\272\004\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000o\227\324h\037\274\005\357\000\342\004\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\013\334xs\030\003\004\307\020\t\005\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000.q\255\013\3151\351\307 0\005\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\354\245\221Y\251\310\304P0W\005\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\214\236\277&\322\307p\262@~\005\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000H\030.^?\217xsP\245\005\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\312\25426qy\327\030`\314\005\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\377\220\030u-S\213\001p\363\005\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\226\005\245A\010\275\244\303\200\032\006\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\207\200Z\017\210\2270X\220A\006\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\231<\003U+f\333\206\240h\006\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\356\326TOJ\243\363|\260\217\006\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\273\310.I\r\253w\004\300\266\006\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\260v\351\003\245VP\313\320\335\006\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\214\341p>w\252\234.\340\004\007\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000\311\366\005\177\2104B\333\360+\007\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000&\203\322\t\243\323\362\363\000S\007\000\000\000\000\000\020\'\000\000\000\000\000\000\001\007\000e\007\222\002Tf\352\177\020z\007\000\000\000\000\000\020\'\000\000\000\000\000\000" indirect_data: "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 I0408 18:33:44.557595 10533 ts_tablet_manager.cc:955] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.013s I0408 18:33:44.557732 10533 tablet_replica.cc:262] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: stopping tablet replica I0408 18:33:44.557916 10533 raft_consensus.cc:2012] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Raft consensus shutting down. I0408 18:33:44.557955 10533 raft_consensus.cc:2039] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Raft consensus is shut down! I0408 18:33:44.583900 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 10518 W0408 18:33:44.586882 10307 connection.cc:513] server connection from 127.6.54.1:38323 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:44.586887 10298 connection.cc:513] client connection to 127.6.54.1:44013 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:44.587105 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:44013 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=36947 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:44093 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:44.619846 10643 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:44.620002 10643 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:44.620733 10643 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:44.620760 10643 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:44.621706 10643 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:44013 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=36947 --tserver_master_addrs=127.6.54.62:44093 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10643 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:44.622686 10643 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:44.622922 10643 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31011us I0408 18:33:44.623070 10643 fs_manager.cc:260] Metadata directory not provided I0408 18:33:44.623109 10643 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:44.624100 10643 fs_manager.cc:397] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s I0408 18:33:44.624348 10643 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:44.624377 10643 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:44.624991 10643 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:44.625366 10643 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.625411 10643 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:44.696760 10643 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:44.696979 10643 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:44.697003 10643 ts_tablet_manager.cc:915] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Loading tablet metadata I0408 18:33:44.697623 10643 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:44.698130 10643 ts_tablet_manager.cc:1080] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:44.698292 10658 ts_tablet_manager.cc:938] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Bootstrapping tablet I0408 18:33:44.698302 10643 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:44.698385 10658 tablet_bootstrap.cc:436] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Bootstrap starting. I0408 18:33:44.698904 10658 tablet_bootstrap.cc:645] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Previous recovery directory found at /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/e8251b7be8b2495cb7a14bb6f3d6559c.recovery: Replaying log files from this location instead of /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/e8251b7be8b2495cb7a14bb6f3d6559c I0408 18:33:44.698935 10658 tablet_bootstrap.cc:652] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Deleting old log files from previous recovery attempt in /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/wal/wals/e8251b7be8b2495cb7a14bb6f3d6559c I0408 18:33:44.705399 10643 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:44013 I0408 18:33:44.705502 10643 webserver.cc:173] Starting webserver on 127.6.54.1:36947 I0408 18:33:44.705521 10643 webserver.cc:184] Document root disabled I0408 18:33:44.705840 10643 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:36947/ I0408 18:33:44.706187 10643 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.BlockManagerType_TsRecoveryITest.TestChangeMaxCellSize_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:44.706631 10764 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:44.706638 10643 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:44.706697 10764 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:44.706802 10764 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:44.708544 10658 log.cc:520] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Log is configured to *not* fsync() on all Append() calls I0408 18:33:44.710403 10763 heartbeater.cc:346] Connected to a master server at 127.6.54.62:44093 I0408 18:33:44.710450 10763 heartbeater.cc:426] Registering TS with master... I0408 18:33:44.710548 10763 heartbeater.cc:475] Master 127.6.54.62:44093 requested a full tablet report, sending... I0408 18:33:44.710855 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 10643 I0408 18:33:44.710968 10321 ts_manager.cc:87] Re-registered known tserver with Master: a8420c2e1fef4bb6b00a09f1d95dd0ec (127.6.54.1:44013) I0408 18:33:44.711722 10321 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:52273 I0408 18:33:44.715873 10658 tablet_bootstrap.cc:436] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0408 18:33:44.716105 10658 tablet_bootstrap.cc:436] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Bootstrap complete. I0408 18:33:44.716769 10658 ts_tablet_manager.cc:955] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.010s I0408 18:33:44.717779 10658 raft_consensus.cc:304] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } } I0408 18:33:44.717825 10658 raft_consensus.cc:605] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8420c2e1fef4bb6b00a09f1d95dd0ec, State: Initialized, Role: FOLLOWER I0408 18:33:44.717941 10658 consensus_queue.cc:226] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } } I0408 18:33:44.718003 10658 raft_consensus.cc:344] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:44.718026 10658 raft_consensus.cc:436] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:44.718093 10658 raft_consensus.cc:2700] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Snoozing failure detection for 1.571s (starting election) I0408 18:33:44.718122 10658 raft_consensus.cc:2752] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:44.718839 10658 raft_consensus.cc:458] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } } I0408 18:33:44.718886 10658 leader_election.cc:258] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:44.719066 10658 ts_tablet_manager.cc:983] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s I0408 18:33:44.719123 10763 heartbeater.cc:467] Master 127.6.54.62:44093 was elected leader, sending a full tablet report... I0408 18:33:44.719343 10774 raft_consensus.cc:2700] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec: Snoozing failure detection for 1.894s (election complete) I0408 18:33:44.719434 10774 raft_consensus.cc:2507] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:44.719467 10774 raft_consensus.cc:572] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [term 2 LEADER]: Becoming Leader. State: Replica: a8420c2e1fef4bb6b00a09f1d95dd0ec, State: Running, Role: LEADER I0408 18:33:44.719547 10774 consensus_queue.cc:203] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } } I0408 18:33:44.719987 10321 catalog_manager.cc:3595] T e8251b7be8b2495cb7a14bb6f3d6559c P a8420c2e1fef4bb6b00a09f1d95dd0ec reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8420c2e1fef4bb6b00a09f1d95dd0ec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 44013 } health_report { overall_health: HEALTHY } } } I0408 18:33:44.921201 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 10643 I0408 18:33:44.924827 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 10299 W0408 18:33:44.927422 10298 connection.cc:513] client connection to 127.6.54.62:44093 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:44.927422 10495 connection.cc:513] client connection to 127.6.54.62:44093 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] BlockManagerType/TsRecoveryITest.TestChangeMaxCellSize/1 (755 ms) [----------] 12 tests from BlockManagerType/TsRecoveryITest (39546 ms total) [----------] 2 tests from DifferentFaultPoints/Kudu969Test [ RUN ] DifferentFaultPoints/Kudu969Test.Test/0 I0408 18:33:44.930035 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:44.963706 10779 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:44.963848 10779 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:44.964601 10779 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:44.964614 10779 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:44.964618 10779 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:44.965626 10779 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.10779 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:44.966495 10779 master_main.cc:73] Initializing master server... I0408 18:33:44.966701 10779 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31011us I0408 18:33:44.966872 10779 fs_manager.cc:260] Metadata directory not provided I0408 18:33:44.966900 10779 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:44.966970 10779 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:44.966994 10779 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:44.968339 10779 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "bd60fa8277e54ef5899a976ca7608958" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.968513 10779 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "bd60fa8277e54ef5899a976ca7608958" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.969763 10779 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:44.969996 10779 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:44.970023 10779 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:44.970752 10779 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:44.970784 10779 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/wal uuid: "bd60fa8277e54ef5899a976ca7608958" format_stamp: "Formatted at 2019-04-08 18:33:44 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:44.970830 10779 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:45.006546 10779 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:45.006621 10779 master_main.cc:76] Starting Master server... I0408 18:33:45.007251 10794 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:45.007270 10794 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:45.007273 10794 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:45.008671 10779 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:36789 I0408 18:33:45.008738 10779 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:45.008750 10779 webserver.cc:184] Document root disabled I0408 18:33:45.008931 10779 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:43565/ I0408 18:33:45.009143 10779 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:45.009593 10838 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:45.011744 10838 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958: Bootstrap starting. I0408 18:33:45.012176 10838 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958: No blocks or log segments found. Creating new log. I0408 18:33:45.012409 10838 log.cc:520] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958: Log is configured to *not* fsync() on all Append() calls I0408 18:33:45.012836 10838 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958: No bootstrap required, opened a new log I0408 18:33:45.013217 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 10779 I0408 18:33:45.013432 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36789 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --maintenance_manager_num_threads=3 with env {} I0408 18:33:45.014199 10838 raft_consensus.cc:304] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60fa8277e54ef5899a976ca7608958" member_type: VOTER } I0408 18:33:45.014237 10838 raft_consensus.cc:330] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:45.014251 10838 raft_consensus.cc:605] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd60fa8277e54ef5899a976ca7608958, State: Initialized, Role: FOLLOWER I0408 18:33:45.014367 10838 consensus_queue.cc:226] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60fa8277e54ef5899a976ca7608958" member_type: VOTER } I0408 18:33:45.014394 10838 raft_consensus.cc:344] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:45.014410 10838 raft_consensus.cc:436] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:45.014438 10838 raft_consensus.cc:2700] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958: Snoozing failure detection for 1.743s (starting election) I0408 18:33:45.014456 10838 raft_consensus.cc:2752] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:45.014904 10838 raft_consensus.cc:458] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60fa8277e54ef5899a976ca7608958" member_type: VOTER } I0408 18:33:45.014973 10838 leader_election.cc:258] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:45.015208 10838 sys_catalog.cc:335] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60fa8277e54ef5899a976ca7608958" member_type: VOTER } } I0408 18:33:45.015221 10842 raft_consensus.cc:2700] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958: Snoozing failure detection for 2.235s (election complete) I0408 18:33:45.015408 10842 raft_consensus.cc:2507] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:45.015307 10838 sys_catalog.cc:338] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [sys.catalog]: This master's current role is: FOLLOWER I0408 18:33:45.016484 10842 raft_consensus.cc:572] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [term 1 LEADER]: Becoming Leader. State: Replica: bd60fa8277e54ef5899a976ca7608958, State: Running, Role: LEADER I0408 18:33:45.016566 10842 consensus_queue.cc:203] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60fa8277e54ef5899a976ca7608958" member_type: VOTER } I0408 18:33:45.016788 10842 sys_catalog.cc:335] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd60fa8277e54ef5899a976ca7608958. Latest consensus state: current_term: 1 leader_uuid: "bd60fa8277e54ef5899a976ca7608958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60fa8277e54ef5899a976ca7608958" member_type: VOTER } } I0408 18:33:45.016881 10842 sys_catalog.cc:338] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [sys.catalog]: This master's current role is: LEADER I0408 18:33:45.016805 10838 sys_catalog.cc:422] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:45.017120 10841 sys_catalog.cc:335] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd60fa8277e54ef5899a976ca7608958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60fa8277e54ef5899a976ca7608958" member_type: VOTER } } I0408 18:33:45.017170 10841 sys_catalog.cc:338] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958 [sys.catalog]: This master's current role is: LEADER I0408 18:33:45.017185 10845 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:45.017381 10779 master_main.cc:79] Master server successfully started. I0408 18:33:45.017629 10845 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0408 18:33:45.047456 10845 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:45.048121 10845 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:45.051650 10845 catalog_manager.cc:3871] T 00000000000000000000000000000000 P bd60fa8277e54ef5899a976ca7608958: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:45.060744 10840 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:45.060940 10840 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:45.061982 10840 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:45.062024 10840 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:45.063583 10840 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36789 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10840 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --maintenance_manager_num_threads=3 --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:45.065052 10840 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:45.065354 10840 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31511us I0408 18:33:45.065548 10840 fs_manager.cc:260] Metadata directory not provided I0408 18:33:45.065593 10840 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:45.065690 10840 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:45.065714 10840 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:45.067626 10840 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "b812540aa2e74153b4f96cc45fc3dd92" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.067847 10840 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "b812540aa2e74153b4f96cc45fc3dd92" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.071449 10840 fs_manager.cc:495] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s I0408 18:33:45.071811 10840 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:45.071851 10840 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:45.072711 10840 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:45.072753 10840 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "b812540aa2e74153b4f96cc45fc3dd92" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.072806 10840 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:45.119283 10840 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:45.119567 10840 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:45.119654 10840 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:45.124233 10840 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:34049 I0408 18:33:45.124318 10840 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:45.124333 10840 webserver.cc:184] Document root disabled I0408 18:33:45.124567 10840 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:42655/ I0408 18:33:45.124800 10840 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:45.125072 10840 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:45.127662 10971 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:45.127684 10971 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:45.127691 10971 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:45.127769 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 10840 I0408 18:33:45.128082 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.6.54.2:0 --local_ip_for_outbound_sockets=127.6.54.2 --webserver_interface=127.6.54.2 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36789 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --maintenance_manager_num_threads=3 with env {} I0408 18:33:45.135947 10970 heartbeater.cc:346] Connected to a master server at 127.6.54.62:36789 I0408 18:33:45.135993 10970 heartbeater.cc:426] Registering TS with master... I0408 18:33:45.136107 10970 heartbeater.cc:475] Master 127.6.54.62:36789 requested a full tablet report, sending... I0408 18:33:45.136607 10805 ts_manager.cc:81] Registered new tserver with Master: b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1:34049) I0408 18:33:45.137362 10805 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:42565 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:45.187227 10974 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:45.187381 10974 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:45.188103 10974 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:45.188134 10974 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.2 I0408 18:33:45.189129 10974 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/info.pb --webserver_interface=127.6.54.2 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36789 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10974 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --maintenance_manager_num_threads=3 --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.2 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:45.190198 10974 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:45.190444 10974 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31511us I0408 18:33:45.190606 10974 fs_manager.cc:260] Metadata directory not provided I0408 18:33:45.190640 10974 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:45.190717 10974 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0408 18:33:45.190735 10974 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:45.192035 10974 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/instance: uuid: "312a97f4d00e453a8e99704e8073fa47" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.192212 10974 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/wal/instance: uuid: "312a97f4d00e453a8e99704e8073fa47" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.193439 10974 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0408 18:33:45.193694 10974 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:45.193727 10974 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:45.194332 10974 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:45.194368 10974 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/wal uuid: "312a97f4d00e453a8e99704e8073fa47" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.194419 10974 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:45.208160 10974 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:45.208365 10974 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:45.208438 10974 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:45.212815 10974 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.2:44415 I0408 18:33:45.212903 10974 webserver.cc:173] Starting webserver on 127.6.54.2:0 I0408 18:33:45.212913 10974 webserver.cc:184] Document root disabled I0408 18:33:45.213129 10974 webserver.cc:311] Webserver started. Bound to: http://127.6.54.2:32933/ I0408 18:33:45.213320 10974 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/info.pb I0408 18:33:45.213673 10974 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:45.214077 11096 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:45.214094 11096 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:45.214097 11096 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:45.215234 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 10974 I0408 18:33:45.215517 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.6.54.3:0 --local_ip_for_outbound_sockets=127.6.54.3 --webserver_interface=127.6.54.3 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36789 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --maintenance_manager_num_threads=3 with env {} I0408 18:33:45.217582 11095 heartbeater.cc:346] Connected to a master server at 127.6.54.62:36789 I0408 18:33:45.217629 11095 heartbeater.cc:426] Registering TS with master... I0408 18:33:45.217717 11095 heartbeater.cc:475] Master 127.6.54.62:36789 requested a full tablet report, sending... I0408 18:33:45.218091 10805 ts_manager.cc:81] Registered new tserver with Master: 312a97f4d00e453a8e99704e8073fa47 (127.6.54.2:44415) I0408 18:33:45.218644 10805 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.2:40763 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:45.248545 11098 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:45.248692 11098 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:45.249351 11098 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:45.249373 11098 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.3 I0408 18:33:45.250331 11098 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/info.pb --webserver_interface=127.6.54.3 --webserver_port=0 --tserver_master_addrs=127.6.54.62:36789 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11098 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --maintenance_manager_num_threads=3 --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.3 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:45.251394 11098 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:45.251652 11098 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 31511us I0408 18:33:45.251770 11098 fs_manager.cc:260] Metadata directory not provided I0408 18:33:45.251794 11098 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:45.251848 11098 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0408 18:33:45.251863 11098 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:45.253130 11098 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/instance: uuid: "8c18cbbe25874ef99a282b8f1bd40dda" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.253288 11098 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/wal/instance: uuid: "8c18cbbe25874ef99a282b8f1bd40dda" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.254478 11098 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s I0408 18:33:45.254701 11098 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:45.254727 11098 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:45.255347 11098 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:45.255378 11098 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/wal uuid: "8c18cbbe25874ef99a282b8f1bd40dda" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:45.255408 11098 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:45.282270 11098 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:45.282460 11098 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:45.282533 11098 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:45.285676 11098 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.3:45705 I0408 18:33:45.285737 11098 webserver.cc:173] Starting webserver on 127.6.54.3:0 I0408 18:33:45.285748 11098 webserver.cc:184] Document root disabled I0408 18:33:45.285934 11098 webserver.cc:311] Webserver started. Bound to: http://127.6.54.3:35367/ I0408 18:33:45.286126 11098 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/info.pb I0408 18:33:45.286576 11098 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:45.286305 11220 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:45.286635 11220 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:45.286648 11220 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:45.289049 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 11098 I0408 18:33:45.290448 11219 heartbeater.cc:346] Connected to a master server at 127.6.54.62:36789 I0408 18:33:45.290493 11219 heartbeater.cc:426] Registering TS with master... I0408 18:33:45.290596 11219 heartbeater.cc:475] Master 127.6.54.62:36789 requested a full tablet report, sending... I0408 18:33:45.290917 10805 ts_manager.cc:81] Registered new tserver with Master: 8c18cbbe25874ef99a282b8f1bd40dda (127.6.54.3:45705) I0408 18:33:45.291503 10805 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.3:50627 I0408 18:33:45.292531 6360 external_mini_cluster.cc:499] 3 TS(s) registered with all masters W0408 18:33:45.299229 10967 generic_service.cc:126] {username='slave'} at 127.0.0.1:42092 forcing change of non-runtime-safe flag flush_threshold_mb I0408 18:33:45.299262 10967 generic_service.cc:146] {username='slave'} at 127.0.0.1:42092 changed flags via RPC: flush_threshold_mb from '1024' to '1' I0408 18:33:45.299515 6360 test_util.cc:200] Using random seed: 328918587 I0408 18:33:45.303673 10805 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33048: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0408 18:33:45.303843 10805 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0408 18:33:45.308780 10906 tablet_service.cc:758] Processing CreateTablet for tablet ad5d88fd99744865a364185c5252c276 (table=test-workload [id=e0476bc956d945dfbd5063b0ae044033]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:45.308962 10906 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet ad5d88fd99744865a364185c5252c276. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:45.309146 11032 tablet_service.cc:758] Processing CreateTablet for tablet ad5d88fd99744865a364185c5252c276 (table=test-workload [id=e0476bc956d945dfbd5063b0ae044033]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:45.309278 11032 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet ad5d88fd99744865a364185c5252c276. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:45.309295 11156 tablet_service.cc:758] Processing CreateTablet for tablet ad5d88fd99744865a364185c5252c276 (table=test-workload [id=e0476bc956d945dfbd5063b0ae044033]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:45.309398 11156 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet ad5d88fd99744865a364185c5252c276. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:45.311080 11156 ts_tablet_manager.cc:1080] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:45.311498 10906 ts_tablet_manager.cc:1080] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:45.311575 11032 ts_tablet_manager.cc:1080] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:45.311803 11240 ts_tablet_manager.cc:938] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Bootstrapping tablet I0408 18:33:45.311887 11240 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Bootstrap starting. I0408 18:33:45.312260 11240 tablet_bootstrap.cc:581] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: No blocks or log segments found. Creating new log. I0408 18:33:45.312467 11240 log.cc:520] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Log is configured to *not* fsync() on all Append() calls I0408 18:33:45.312599 11241 ts_tablet_manager.cc:938] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Bootstrapping tablet I0408 18:33:45.312654 11241 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Bootstrap starting. I0408 18:33:45.312944 11241 tablet_bootstrap.cc:581] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: No blocks or log segments found. Creating new log. I0408 18:33:45.313138 11241 log.cc:520] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Log is configured to *not* fsync() on all Append() calls I0408 18:33:45.313242 11242 ts_tablet_manager.cc:938] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Bootstrapping tablet I0408 18:33:45.313293 11242 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Bootstrap starting. I0408 18:33:45.313602 11242 tablet_bootstrap.cc:581] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: No blocks or log segments found. Creating new log. I0408 18:33:45.313798 11242 log.cc:520] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Log is configured to *not* fsync() on all Append() calls I0408 18:33:45.314260 11241 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: No bootstrap required, opened a new log I0408 18:33:45.314324 11241 ts_tablet_manager.cc:955] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0408 18:33:45.314399 11242 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: No bootstrap required, opened a new log I0408 18:33:45.314457 11242 ts_tablet_manager.cc:955] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:45.314551 11240 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: No bootstrap required, opened a new log I0408 18:33:45.314632 11240 ts_tablet_manager.cc:955] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I0408 18:33:45.315655 11242 raft_consensus.cc:304] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.315697 11242 raft_consensus.cc:330] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:45.315719 11242 raft_consensus.cc:605] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312a97f4d00e453a8e99704e8073fa47, State: Initialized, Role: FOLLOWER I0408 18:33:45.315843 11242 consensus_queue.cc:226] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.315914 11241 raft_consensus.cc:304] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.315986 11095 heartbeater.cc:467] Master 127.6.54.62:36789 was elected leader, sending a full tablet report... I0408 18:33:45.315996 11241 raft_consensus.cc:330] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:45.316020 11241 raft_consensus.cc:605] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b812540aa2e74153b4f96cc45fc3dd92, State: Initialized, Role: FOLLOWER I0408 18:33:45.316155 11241 consensus_queue.cc:226] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.316156 11242 ts_tablet_manager.cc:983] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0408 18:33:45.316177 11240 raft_consensus.cc:304] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.316222 11240 raft_consensus.cc:330] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:45.316352 11240 raft_consensus.cc:605] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c18cbbe25874ef99a282b8f1bd40dda, State: Initialized, Role: FOLLOWER I0408 18:33:45.316377 10970 heartbeater.cc:467] Master 127.6.54.62:36789 was elected leader, sending a full tablet report... I0408 18:33:45.316390 11241 ts_tablet_manager.cc:983] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:45.316520 11240 consensus_queue.cc:226] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.316673 11240 ts_tablet_manager.cc:983] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I0408 18:33:45.316658 11219 heartbeater.cc:467] Master 127.6.54.62:36789 was elected leader, sending a full tablet report... W0408 18:33:45.377889 10971 tablet.cc:1679] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Can't schedule compaction. Clean time has not been advanced past its initial value. I0408 18:33:45.439677 11247 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0408 18:33:45.439769 11247 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Snoozing failure detection for 1.940s (starting election) I0408 18:33:45.439846 11247 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.440075 11247 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:45.440158 11247 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8c18cbbe25874ef99a282b8f1bd40dda I0408 18:33:45.442823 11169 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "b812540aa2e74153b4f96cc45fc3dd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" is_pre_election: true I0408 18:33:45.442898 11169 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 2.015s (vote granted) I0408 18:33:45.442925 11169 raft_consensus.cc:2215] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b812540aa2e74153b4f96cc45fc3dd92 in term 0. I0408 18:33:45.442956 11052 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "b812540aa2e74153b4f96cc45fc3dd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312a97f4d00e453a8e99704e8073fa47" is_pre_election: true I0408 18:33:45.443025 11052 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 1.550s (vote granted) I0408 18:33:45.443047 11052 raft_consensus.cc:2215] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b812540aa2e74153b4f96cc45fc3dd92 in term 0. I0408 18:33:45.443192 10862 leader_election.cc:363] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8c18cbbe25874ef99a282b8f1bd40dda I0408 18:33:45.443219 10862 leader_election.cc:258] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0408 18:33:45.443274 10859 leader_election.cc:363] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:45.443287 11247 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Snoozing failure detection for 2.236s (election complete) I0408 18:33:45.443305 11247 raft_consensus.cc:2507] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0408 18:33:45.443329 11247 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0408 18:33:45.443343 11247 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Snoozing failure detection for 1.881s (starting election) I0408 18:33:45.443351 11247 raft_consensus.cc:2752] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:45.443961 11247 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.444103 11247 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 election: Requesting vote from peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:45.444135 11247 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 election: Requesting vote from peer 8c18cbbe25874ef99a282b8f1bd40dda I0408 18:33:45.444386 11052 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "b812540aa2e74153b4f96cc45fc3dd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312a97f4d00e453a8e99704e8073fa47" I0408 18:33:45.444396 11169 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "b812540aa2e74153b4f96cc45fc3dd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" I0408 18:33:45.444414 11169 raft_consensus.cc:2752] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:45.444413 11052 raft_consensus.cc:2752] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:45.444432 11169 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 1.573s (vote granted) I0408 18:33:45.444442 11052 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 2.220s (vote granted) I0408 18:33:45.444926 11052 raft_consensus.cc:2215] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b812540aa2e74153b4f96cc45fc3dd92 in term 1. I0408 18:33:45.444926 11169 raft_consensus.cc:2215] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b812540aa2e74153b4f96cc45fc3dd92 in term 1. I0408 18:33:45.445120 10859 leader_election.cc:363] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 election: Vote granted by peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:45.445143 10859 leader_election.cc:258] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:45.445233 10862 leader_election.cc:363] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 1 election: Vote granted by peer 8c18cbbe25874ef99a282b8f1bd40dda I0408 18:33:45.445251 11247 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Snoozing failure detection for 1.508s (election complete) I0408 18:33:45.445266 11247 raft_consensus.cc:2507] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:45.445346 11247 raft_consensus.cc:572] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 LEADER]: Becoming Leader. State: Replica: b812540aa2e74153b4f96cc45fc3dd92, State: Running, Role: LEADER I0408 18:33:45.445433 11247 consensus_queue.cc:203] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:45.446271 10805 catalog_manager.cc:3595] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 reported cstate change: term changed from 0 to 1, leader changed from to b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1). New cstate: current_term: 1 leader_uuid: "b812540aa2e74153b4f96cc45fc3dd92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } health_report { overall_health: HEALTHY } } } W0408 18:33:45.454386 10967 generic_service.cc:126] {username='slave'} at 127.0.0.1:42092 forcing change of non-runtime-safe flag fault_crash_before_flush_tablet_meta_after_compaction I0408 18:33:45.454429 10967 generic_service.cc:146] {username='slave'} at 127.0.0.1:42092 changed flags via RPC: fault_crash_before_flush_tablet_meta_after_compaction from '0' to '0.3' I0408 18:33:45.463133 11169 raft_consensus.cc:1058] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 1 FOLLOWER]: Refusing update from remote peer b812540aa2e74153b4f96cc45fc3dd92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0408 18:33:45.463191 11052 raft_consensus.cc:1058] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 1 FOLLOWER]: Refusing update from remote peer b812540aa2e74153b4f96cc45fc3dd92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0408 18:33:45.463477 11247 consensus_queue.cc:962] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0408 18:33:45.463580 11250 consensus_queue.cc:962] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0408 18:33:45.464184 11096 tablet.cc:1679] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Can't schedule compaction. Clean time has not been advanced past its initial value. I0408 18:33:46.128438 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling FlushMRSOp(ad5d88fd99744865a364185c5252c276): perf score=0.945090 I0408 18:33:46.128648 10856 tablet.cc:1153] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:46.128666 10856 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:46.128680 10856 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:46.128685 10856 tablet.cc:1155] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Memstore in-memory size: 2039575 bytes I0408 18:33:46.128723 10856 tablet.cc:1444] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249553414885376 or (T in {6368249553414885376})}] I0408 18:33:46.130087 10856 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:46.145107 10856 tablet.cc:1529] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:46.147220 10856 tablet.cc:1584] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:46.147305 10856 tablet.cc:1586] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249553490821120 or (T in {6368249553490821120})}] I0408 18:33:46.148993 10856 delta_tracker.cc:722] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flushing 14 deltas (763 bytes in memory) from DMS 0 I0408 18:33:46.151371 10856 delta_tracker.cc:663] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flushed delta block 0000000000000008 (405 bytes on disk) stats: ts range=[6368249553418129408, 6368249553493798912], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:14] I0408 18:33:46.153383 10856 tablet.cc:1634] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush successful on 14100 rows (68130 bytes) I0408 18:33:46.153630 10856 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: FlushMRSOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.025s user 0.016s sys 0.000s Metrics: {"cfile_init":1,"compiler_manager_pool.queue_time_us":534,"data dirs.queue_time_us":98,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":881,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":42,"thread_start_us":107,"threads_started":2} I0408 18:33:46.153851 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276): 130558 bytes on disk I0408 18:33:46.153898 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276): 130558 bytes on disk I0408 18:33:46.153926 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276): 130558 bytes on disk I0408 18:33:46.154100 10855 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4} I0408 18:33:46.154144 10855 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:46.154167 10855 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:46.904968 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling FlushMRSOp(ad5d88fd99744865a364185c5252c276): perf score=0.945090 I0408 18:33:46.905123 10856 tablet.cc:1153] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:46.905146 10856 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:46.905150 10856 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:46.905153 10856 tablet.cc:1155] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Memstore in-memory size: 2039575 bytes I0408 18:33:46.905160 10856 tablet.cc:1444] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249556596342784 or (T in {6368249556596342784})}] I0408 18:33:46.905323 10856 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:46.913126 10856 tablet.cc:1529] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:46.913223 10856 tablet.cc:1584] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:46.913272 10856 tablet.cc:1586] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249556627857408 or (T in {6368249556627857408})}] I0408 18:33:46.914090 10856 delta_tracker.cc:722] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flushing 6 deltas (763 bytes in memory) from DMS 0 I0408 18:33:46.914255 10856 delta_tracker.cc:663] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flushed delta block 0000000000000015 (281 bytes on disk) stats: ts range=[6368249556609642496, 6368249556627238912], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:6] I0408 18:33:46.915746 10856 tablet.cc:1634] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flush successful on 15000 rows (58691 bytes) I0408 18:33:46.915872 10856 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: FlushMRSOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.011s user 0.008s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":891,"data dirs.run_cpu_time_us":152,"data dirs.run_wall_time_us":848,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":417,"lbm_writes_lt_1ms":42,"thread_start_us":57,"threads_started":1} I0408 18:33:46.915983 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276): 127248 bytes on disk I0408 18:33:46.916035 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling CompactRowSetsOp(ad5d88fd99744865a364185c5252c276): perf score=0.989928 I0408 18:33:46.916083 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276): 127248 bytes on disk I0408 18:33:46.916134 10855 tablet.cc:1664] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:46.916146 10855 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:46.916152 10855 compaction.cc:917] RowSet(0)(current size on disk: ~199093 bytes) I0408 18:33:46.916155 10855 compaction.cc:917] RowSet(1)(current size on disk: ~186220 bytes) I0408 18:33:46.916160 10855 tablet.cc:1444] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249556637298688 or (T in {6368249556637298688})}] I0408 18:33:46.916167 10854 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4} I0408 18:33:46.916230 10854 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:46.916301 10855 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:46.916304 10971 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling CompactRowSetsOp(ad5d88fd99744865a364185c5252c276): perf score=0.989928 I0408 18:33:46.916366 10854 tablet.cc:1664] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: stage 1 complete, picked 0 rowsets to compact or flush I0408 18:33:46.916376 10854 compaction.cc:914] Selected 0 rowsets to compact: I0408 18:33:46.916384 10854 tablet.cc:1444] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249556637298688 or (T in {6368249556637298688})}] I0408 18:33:46.916968 10854 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:46.917503 10854 tablet.cc:1473] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets. I0408 18:33:46.919224 10854 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: CompactRowSetsOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"data dirs.queue_time_us":2620,"data dirs.run_cpu_time_us":180,"data dirs.run_wall_time_us":1464,"lbm_write_time_us":38,"lbm_writes_lt_1ms":6} I0408 18:33:46.930541 10855 tablet.cc:1529] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:46.930590 10855 tablet.cc:1584] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:46.930598 10855 tablet.cc:1586] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249556698275840 or (T in {6368249556698275840})}] I0408 18:33:46.934176 10855 delta_tracker.cc:722] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flushing 22 deltas (3751 bytes in memory) from DMS 0 I0408 18:33:46.934428 10855 delta_tracker.cc:663] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Flushed delta block 0000000000000028 (547 bytes on disk) stats: ts range=[6368249556646612992, 6368249556715175936], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:22] W0408 18:33:46.934509 10855 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:46.934628 10855 fault_injection.cc:42] THIS SERVER MAY CRASH! E0408 18:33:46.934656 10855 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_flush_tablet_meta_after_compaction (process will exit) W0408 18:33:46.939334 11108 connection.cc:513] server connection from 127.6.54.1:49825 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:46.939376 11253 connection.cc:513] client connection to 127.6.54.1:34049 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:46.939404 11253 meta_cache.cc:219] Tablet ad5d88fd99744865a364185c5252c276: Replica b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1:34049) has failed: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:46.939385 10984 connection.cc:513] server connection from 127.6.54.1:60027 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:46.939445 10788 connection.cc:513] client connection to 127.6.54.1:34049 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:46.939561 11253 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0408 18:33:46.939760 10788 connection.cc:513] server connection from 127.6.54.1:42565 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:46.939924 10778 connection.cc:513] client connection to 127.6.54.1:34049 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:47.944757 11253 batcher.cc:349] Timed out: Failed to write batch of 100 ops to tablet ad5d88fd99744865a364185c5252c276 after 41 attempt(s): Failed to write to server: (no server available): Write(tablet: ad5d88fd99744865a364185c5252c276, num_ops: 100, num_attempts: 41) passed its deadline: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:47.944954 6360 ts_recovery-itest.cc:752] could not flush session: IO error: Some errors occurred I0408 18:33:47.945020 6360 ts_recovery-itest.cc:756] successfully detected TS crash! W0408 18:33:48.010730 11253 meta_cache.cc:219] Tablet ad5d88fd99744865a364185c5252c276: Replica b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1:34049) has failed: Network error: Client connection negotiation failed: client connection to 127.6.54.1:34049: connect: Connection refused (error 111) I0408 18:33:48.434486 11289 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b812540aa2e74153b4f96cc45fc3dd92) I0408 18:33:48.434541 11289 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 2.184s (starting election) I0408 18:33:48.434612 11289 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:48.434859 11289 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c18cbbe25874ef99a282b8f1bd40dda I0408 18:33:48.434916 11289 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b812540aa2e74153b4f96cc45fc3dd92 I0408 18:33:48.435300 11292 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b812540aa2e74153b4f96cc45fc3dd92) W0408 18:33:48.435304 10985 leader_election.cc:287] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b812540aa2e74153b4f96cc45fc3dd92: Network error: Client connection negotiation failed: client connection to 127.6.54.1:34049: connect: Connection refused (error 111) I0408 18:33:48.435334 11292 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 1.624s (starting election) I0408 18:33:48.435386 11292 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:48.435572 11292 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:48.435628 11292 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b812540aa2e74153b4f96cc45fc3dd92 W0408 18:33:48.435904 11109 leader_election.cc:287] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b812540aa2e74153b4f96cc45fc3dd92: Network error: Client connection negotiation failed: client connection to 127.6.54.1:34049: connect: Connection refused (error 111) I0408 18:33:48.437973 11169 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "312a97f4d00e453a8e99704e8073fa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 2502 } } ignore_live_leader: false dest_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" is_pre_election: true I0408 18:33:48.438004 11169 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 1.971s (vote granted) I0408 18:33:48.438017 11169 raft_consensus.cc:2215] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 312a97f4d00e453a8e99704e8073fa47 in term 1. I0408 18:33:48.438098 11052 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" candidate_term: 2 candidate_status { last_received { term: 1 index: 2502 } } ignore_live_leader: false dest_uuid: "312a97f4d00e453a8e99704e8073fa47" is_pre_election: true I0408 18:33:48.438132 11052 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 1.628s (vote granted) I0408 18:33:48.438144 11052 raft_consensus.cc:2215] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c18cbbe25874ef99a282b8f1bd40dda in term 1. I0408 18:33:48.438215 10987 leader_election.cc:363] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8c18cbbe25874ef99a282b8f1bd40dda I0408 18:33:48.438258 10987 leader_election.cc:258] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0408 18:33:48.438310 11289 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 1.865s (election complete) I0408 18:33:48.438329 11108 leader_election.cc:363] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 pre-election: Vote granted by peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:48.438335 11289 raft_consensus.cc:2507] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0408 18:33:48.438344 11108 leader_election.cc:258] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0408 18:33:48.438345 11289 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b812540aa2e74153b4f96cc45fc3dd92) I0408 18:33:48.438354 11289 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 1.564s (starting election) I0408 18:33:48.438360 11289 raft_consensus.cc:2752] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:48.438941 11289 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:48.439072 11289 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 election: Requesting vote from peer 8c18cbbe25874ef99a282b8f1bd40dda I0408 18:33:48.439113 11289 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 election: Requesting vote from peer b812540aa2e74153b4f96cc45fc3dd92 I0408 18:33:48.439267 11292 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 1.840s (election complete) I0408 18:33:48.439293 11292 raft_consensus.cc:2507] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 1 FOLLOWER]: Leader pre-election won for term 2 I0408 18:33:48.439301 11292 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 1 FOLLOWER]: Starting leader election (detected failure of leader b812540aa2e74153b4f96cc45fc3dd92) I0408 18:33:48.439306 11292 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 1.968s (starting election) I0408 18:33:48.439311 11292 raft_consensus.cc:2752] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:48.439366 11169 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "312a97f4d00e453a8e99704e8073fa47" candidate_term: 2 candidate_status { last_received { term: 1 index: 2502 } } ignore_live_leader: false dest_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" W0408 18:33:48.439404 10985 leader_election.cc:287] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b812540aa2e74153b4f96cc45fc3dd92: Network error: Client connection negotiation failed: client connection to 127.6.54.1:34049: connect: Connection refused (error 111) I0408 18:33:48.439819 11292 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:48.439960 11292 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 election: Requesting vote from peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:48.439994 11169 raft_consensus.cc:2145] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 312a97f4d00e453a8e99704e8073fa47 in current term 2: Already voted for candidate 8c18cbbe25874ef99a282b8f1bd40dda in this term. I0408 18:33:48.439999 11292 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 election: Requesting vote from peer b812540aa2e74153b4f96cc45fc3dd92 I0408 18:33:48.440214 10987 leader_election.cc:377] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 election: Vote denied by peer 8c18cbbe25874ef99a282b8f1bd40dda. Message: Invalid argument: T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 312a97f4d00e453a8e99704e8073fa47 in current term 2: Already voted for candidate 8c18cbbe25874ef99a282b8f1bd40dda in this term. W0408 18:33:48.440225 11109 leader_election.cc:287] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b812540aa2e74153b4f96cc45fc3dd92: Network error: Client connection negotiation failed: client connection to 127.6.54.1:34049: connect: Connection refused (error 111) I0408 18:33:48.440239 10987 leader_election.cc:258] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0408 18:33:48.440280 11289 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 2.178s (election complete) I0408 18:33:48.440292 11052 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" candidate_term: 2 candidate_status { last_received { term: 1 index: 2502 } } ignore_live_leader: false dest_uuid: "312a97f4d00e453a8e99704e8073fa47" I0408 18:33:48.440295 11289 raft_consensus.cc:2455] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0408 18:33:48.440328 11052 raft_consensus.cc:2145] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c18cbbe25874ef99a282b8f1bd40dda in current term 2: Already voted for candidate 312a97f4d00e453a8e99704e8073fa47 in this term. I0408 18:33:48.440551 11108 leader_election.cc:377] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 election: Vote denied by peer 312a97f4d00e453a8e99704e8073fa47. Message: Invalid argument: T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c18cbbe25874ef99a282b8f1bd40dda in current term 2: Already voted for candidate 312a97f4d00e453a8e99704e8073fa47 in this term. I0408 18:33:48.440569 11108 leader_election.cc:258] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0408 18:33:48.440627 11292 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 1.673s (election complete) I0408 18:33:48.440649 11292 raft_consensus.cc:2455] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority W0408 18:33:48.945183 6360 thread.cc:454] Waited for 1000ms trying to join with updater (tid 11257) W0408 18:33:48.945746 11252 batcher.cc:349] Timed out: Failed to write batch of 1 ops to tablet ad5d88fd99744865a364185c5252c276 after 2 attempt(s): Failed to write to server: 312a97f4d00e453a8e99704e8073fa47 (127.6.54.2:44415): Write RPC to 127.6.54.2:44415 timed out after 2.000s (SENT) W0408 18:33:48.945816 11259 ts_recovery-itest.cc:666] failed to flush updates: IO error: Some errors occurred W0408 18:33:48.953192 11260 ts_recovery-itest.cc:666] failed to flush updates: IO error: Some errors occurred W0408 18:33:48.960407 11257 ts_recovery-itest.cc:666] failed to flush updates: IO error: Some errors occurred W0408 18:33:48.978605 11258 ts_recovery-itest.cc:666] failed to flush updates: IO error: Some errors occurred I0408 18:33:48.979230 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:34049 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=42655 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:36789 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --maintenance_manager_num_threads=3 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:49.013936 11297 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:49.014096 11297 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:49.014830 11297 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:49.014861 11297 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:49.015909 11297 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:34049 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=42655 --tserver_master_addrs=127.6.54.62:36789 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11297 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --maintenance_manager_num_threads=3 --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:49.016993 11297 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:49.017246 11297 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 33511us I0408 18:33:49.017391 11297 fs_manager.cc:260] Metadata directory not provided I0408 18:33:49.017442 11297 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:49.018637 11297 fs_manager.cc:397] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I0408 18:33:49.018844 11297 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:49.018874 11297 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:49.020774 11297 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0408 18:33:49.021167 11297 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "b812540aa2e74153b4f96cc45fc3dd92" format_stamp: "Formatted at 2019-04-08 18:33:45 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:49.021219 11297 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 18 Total live bytes: 793865 Total live bytes (after alignment): 847872 Total number of LBM containers: 12 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:49.053100 11297 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:49.053313 11297 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:49.053335 11297 ts_tablet_manager.cc:915] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Loading tablet metadata I0408 18:33:49.053984 11297 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:49.054498 11297 ts_tablet_manager.cc:1080] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:49.054757 11314 ts_tablet_manager.cc:938] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Bootstrapping tablet I0408 18:33:49.054755 11297 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:49.054863 11314 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Bootstrap starting. I0408 18:33:49.058995 11297 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:34049 I0408 18:33:49.059088 11297 webserver.cc:173] Starting webserver on 127.6.54.1:42655 I0408 18:33:49.059110 11297 webserver.cc:184] Document root disabled I0408 18:33:49.059592 11297 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:42655/ I0408 18:33:49.059798 11297 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:49.060118 11297 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:49.060503 11420 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:49.060534 11420 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:49.060540 11420 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:49.063339 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 11297 I0408 18:33:49.067198 11419 heartbeater.cc:346] Connected to a master server at 127.6.54.62:36789 I0408 18:33:49.067239 11419 heartbeater.cc:426] Registering TS with master... I0408 18:33:49.067342 11419 heartbeater.cc:475] Master 127.6.54.62:36789 requested a full tablet report, sending... I0408 18:33:49.067808 10805 ts_manager.cc:87] Re-registered known tserver with Master: b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1:34049) I0408 18:33:49.068387 10805 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:39843 I0408 18:33:49.086220 11314 log.cc:520] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Log is configured to *not* fsync() on all Append() calls I0408 18:33:49.250574 11314 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2503 overwritten=0 applied=2502 ignored=1701} inserts{seen=400 ignored=0} mutations{seen=796 ignored=0} orphaned_commits=0. Pending: 1 replicates I0408 18:33:49.250895 11314 tablet_bootstrap.cc:436] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Bootstrap complete. I0408 18:33:49.251722 11314 ts_tablet_manager.cc:955] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Time spent bootstrapping tablet: real 0.197s user 0.160s sys 0.035s I0408 18:33:49.252665 11314 raft_consensus.cc:304] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:49.253010 11314 raft_consensus.cc:605] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b812540aa2e74153b4f96cc45fc3dd92, State: Initialized, Role: FOLLOWER I0408 18:33:49.253134 11314 consensus_queue.cc:226] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2502, Last appended: 1.2503, Last appended by leader: 2503, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:49.253290 11419 heartbeater.cc:467] Master 127.6.54.62:36789 was elected leader, sending a full tablet report... I0408 18:33:49.253298 11314 ts_tablet_manager.cc:983] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- bd60fa8277e54ef5899a976ca7608958 | 127.6.54.62:36789 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 312a97f4d00e453a8e99704e8073fa47 | 127.6.54.2:44415 | HEALTHY 8c18cbbe25874ef99a282b8f1bd40dda | 127.6.54.3:45705 | HEALTHY b812540aa2e74153b4f96cc45fc3dd92 | 127.6.54.1:34049 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.6.54.1 | experimental | 127.6.54.1:34049 local_ip_for_outbound_sockets | 127.6.54.2 | experimental | 127.6.54.2:44415 local_ip_for_outbound_sockets | 127.6.54.3 | experimental | 127.6.54.3:45705 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb | hidden | 127.6.54.1:34049 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/info.pb | hidden | 127.6.54.2:44415 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/info.pb | hidden | 127.6.54.3:45705 Version Summary Version | Servers --------------------------+------------------------- 1.7.0-cdh5.17.0-SNAPSHOT | all 4 server(s) checked Tablet ad5d88fd99744865a364185c5252c276 of table 'test-workload' is conflicted: Tablet ad5d88fd99744865a364185c5252c276 of table 'test-workload' replicas' active configs disagree with the master's 312a97f4d00e453a8e99704e8073fa47 (127.6.54.2:44415): RUNNING 8c18cbbe25874ef99a282b8f1bd40dda (127.6.54.3:45705): RUNNING b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1:34049): RUNNING [LEADER] All reported replicas are: A = 312a97f4d00e453a8e99704e8073fa47 B = 8c18cbbe25874ef99a282b8f1bd40dda C = b812540aa2e74153b4f96cc45fc3dd92 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | A B C | 2 | -1 | Yes C | A B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0408 18:33:49.283196 6360 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0408 18:33:49.310884 11420 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling CompactRowSetsOp(ad5d88fd99744865a364185c5252c276): perf score=0.989928 I0408 18:33:49.310992 11420 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276): 257806 bytes on disk I0408 18:33:49.311115 11304 tablet.cc:1664] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:49.311144 11304 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:49.311172 11304 compaction.cc:917] RowSet(0)(current size on disk: ~199093 bytes) I0408 18:33:49.311180 11304 compaction.cc:917] RowSet(1)(current size on disk: ~186220 bytes) I0408 18:33:49.311213 11304 tablet.cc:1444] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249556718583808 or (T in {6368249556718583808})}] I0408 18:33:49.311295 11303 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":24,"lbm_reads_lt_1ms":8} I0408 18:33:49.311364 11420 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling CompactRowSetsOp(ad5d88fd99744865a364185c5252c276): perf score=0.989928 I0408 18:33:49.311409 11303 tablet.cc:1664] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: stage 1 complete, picked 0 rowsets to compact or flush I0408 18:33:49.311424 11303 compaction.cc:914] Selected 0 rowsets to compact: I0408 18:33:49.311434 11303 tablet.cc:1444] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249556718583808 or (T in {6368249556718583808})}] I0408 18:33:49.311655 11303 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:49.311669 11304 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:49.311938 11303 tablet.cc:1473] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets. I0408 18:33:49.313355 11303 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: CompactRowSetsOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"data dirs.queue_time_us":971,"data dirs.run_cpu_time_us":160,"data dirs.run_wall_time_us":920,"lbm_write_time_us":75,"lbm_writes_lt_1ms":6,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":114} I0408 18:33:49.324216 11304 tablet.cc:1529] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:49.324256 11304 tablet.cc:1584] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:49.324263 11304 tablet.cc:1586] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249556718583808 or (T in {6368249556718583808})}] I0408 18:33:49.328032 11304 tablet.cc:1634] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Compaction successful on 29100 rows (137933 bytes) I0408 18:33:49.328177 11304 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: CompactRowSetsOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"cfile_cache_hit":16,"cfile_cache_hit_bytes":11753,"cfile_cache_miss":56,"cfile_cache_miss_bytes":608561,"cfile_init":5,"data dirs.queue_time_us":35,"data dirs.run_cpu_time_us":89,"data dirs.run_wall_time_us":709,"delta_iterators_relevant":8,"lbm_read_time_us":361,"lbm_reads_lt_1ms":76,"lbm_write_time_us":564,"lbm_writes_lt_1ms":54} I0408 18:33:49.328266 11420 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276): 270913 bytes on disk I0408 18:33:49.328313 11420 maintenance_manager.cc:302] P b812540aa2e74153b4f96cc45fc3dd92: Scheduling UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276): 270913 bytes on disk I0408 18:33:49.328399 11304 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4} I0408 18:33:49.328418 11303 maintenance_manager.cc:492] P b812540aa2e74153b4f96cc45fc3dd92: UndoDeltaBlockGCOp(ad5d88fd99744865a364185c5252c276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- bd60fa8277e54ef5899a976ca7608958 | 127.6.54.62:36789 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 312a97f4d00e453a8e99704e8073fa47 | 127.6.54.2:44415 | HEALTHY 8c18cbbe25874ef99a282b8f1bd40dda | 127.6.54.3:45705 | HEALTHY b812540aa2e74153b4f96cc45fc3dd92 | 127.6.54.1:34049 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.6.54.1 | experimental | 127.6.54.1:34049 local_ip_for_outbound_sockets | 127.6.54.2 | experimental | 127.6.54.2:44415 local_ip_for_outbound_sockets | 127.6.54.3 | experimental | 127.6.54.3:45705 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb | hidden | 127.6.54.1:34049 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/info.pb | hidden | 127.6.54.2:44415 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/info.pb | hidden | 127.6.54.3:45705 Version Summary Version | Servers --------------------------+------------------------- 1.7.0-cdh5.17.0-SNAPSHOT | all 4 server(s) checked Tablet ad5d88fd99744865a364185c5252c276 of table 'test-workload' is conflicted: Tablet ad5d88fd99744865a364185c5252c276 of table 'test-workload' replicas' active configs disagree with the master's 312a97f4d00e453a8e99704e8073fa47 (127.6.54.2:44415): RUNNING 8c18cbbe25874ef99a282b8f1bd40dda (127.6.54.3:45705): RUNNING b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1:34049): RUNNING [LEADER] All reported replicas are: A = 312a97f4d00e453a8e99704e8073fa47 B = 8c18cbbe25874ef99a282b8f1bd40dda C = b812540aa2e74153b4f96cc45fc3dd92 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | A B C | 2 | -1 | Yes C | A B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0408 18:33:49.453959 6360 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- bd60fa8277e54ef5899a976ca7608958 | 127.6.54.62:36789 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 312a97f4d00e453a8e99704e8073fa47 | 127.6.54.2:44415 | HEALTHY 8c18cbbe25874ef99a282b8f1bd40dda | 127.6.54.3:45705 | HEALTHY b812540aa2e74153b4f96cc45fc3dd92 | 127.6.54.1:34049 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.6.54.1 | experimental | 127.6.54.1:34049 local_ip_for_outbound_sockets | 127.6.54.2 | experimental | 127.6.54.2:44415 local_ip_for_outbound_sockets | 127.6.54.3 | experimental | 127.6.54.3:45705 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb | hidden | 127.6.54.1:34049 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/info.pb | hidden | 127.6.54.2:44415 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/info.pb | hidden | 127.6.54.3:45705 Version Summary Version | Servers --------------------------+------------------------- 1.7.0-cdh5.17.0-SNAPSHOT | all 4 server(s) checked Tablet ad5d88fd99744865a364185c5252c276 of table 'test-workload' is conflicted: Tablet ad5d88fd99744865a364185c5252c276 of table 'test-workload' replicas' active configs disagree with the master's 312a97f4d00e453a8e99704e8073fa47 (127.6.54.2:44415): RUNNING 8c18cbbe25874ef99a282b8f1bd40dda (127.6.54.3:45705): RUNNING b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1:34049): RUNNING [LEADER] All reported replicas are: A = 312a97f4d00e453a8e99704e8073fa47 B = 8c18cbbe25874ef99a282b8f1bd40dda C = b812540aa2e74153b4f96cc45fc3dd92 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | A B C | 2 | -1 | Yes C | A B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0408 18:33:49.699820 6360 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- bd60fa8277e54ef5899a976ca7608958 | 127.6.54.62:36789 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 312a97f4d00e453a8e99704e8073fa47 | 127.6.54.2:44415 | HEALTHY 8c18cbbe25874ef99a282b8f1bd40dda | 127.6.54.3:45705 | HEALTHY b812540aa2e74153b4f96cc45fc3dd92 | 127.6.54.1:34049 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.6.54.1 | experimental | 127.6.54.1:34049 local_ip_for_outbound_sockets | 127.6.54.2 | experimental | 127.6.54.2:44415 local_ip_for_outbound_sockets | 127.6.54.3 | experimental | 127.6.54.3:45705 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb | hidden | 127.6.54.1:34049 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/info.pb | hidden | 127.6.54.2:44415 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/info.pb | hidden | 127.6.54.3:45705 Version Summary Version | Servers --------------------------+------------------------- 1.7.0-cdh5.17.0-SNAPSHOT | all 4 server(s) checked Tablet ad5d88fd99744865a364185c5252c276 of table 'test-workload' is conflicted: Tablet ad5d88fd99744865a364185c5252c276 of table 'test-workload' replicas' active configs disagree with the master's 312a97f4d00e453a8e99704e8073fa47 (127.6.54.2:44415): RUNNING 8c18cbbe25874ef99a282b8f1bd40dda (127.6.54.3:45705): RUNNING b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1:34049): RUNNING [LEADER] All reported replicas are: A = 312a97f4d00e453a8e99704e8073fa47 B = 8c18cbbe25874ef99a282b8f1bd40dda C = b812540aa2e74153b4f96cc45fc3dd92 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | A B C | 2 | -1 | Yes C | A B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0408 18:33:50.059295 6360 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0408 18:33:50.114326 11511 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0408 18:33:50.114379 11511 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 2.043s (starting election) I0408 18:33:50.114460 11511 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:50.114662 11511 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:50.114717 11511 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer b812540aa2e74153b4f96cc45fc3dd92 I0408 18:33:50.115056 11052 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" candidate_term: 3 candidate_status { last_received { term: 1 index: 2502 } } ignore_live_leader: false dest_uuid: "312a97f4d00e453a8e99704e8073fa47" is_pre_election: true I0408 18:33:50.115116 11052 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 1.702s (vote granted) I0408 18:33:50.115133 11052 raft_consensus.cc:2215] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c18cbbe25874ef99a282b8f1bd40dda in term 2. I0408 18:33:50.115363 11108 leader_election.cc:363] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 pre-election: Vote granted by peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:50.115387 11108 leader_election.cc:258] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0408 18:33:50.115664 11511 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 2.349s (election complete) I0408 18:33:50.115691 11511 raft_consensus.cc:2507] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Leader pre-election won for term 3 I0408 18:33:50.115700 11511 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0408 18:33:50.115707 11511 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 2.180s (starting election) I0408 18:33:50.115713 11511 raft_consensus.cc:2752] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 2 FOLLOWER]: Advancing to term 3 I0408 18:33:50.116338 11511 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:50.116490 11511 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 election: Requesting vote from peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:50.116542 11511 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 election: Requesting vote from peer b812540aa2e74153b4f96cc45fc3dd92 I0408 18:33:50.116801 11052 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" candidate_term: 3 candidate_status { last_received { term: 1 index: 2502 } } ignore_live_leader: false dest_uuid: "312a97f4d00e453a8e99704e8073fa47" I0408 18:33:50.116830 11052 raft_consensus.cc:2752] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 2 FOLLOWER]: Advancing to term 3 I0408 18:33:50.116845 11052 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47: Snoozing failure detection for 3.194s (vote granted) I0408 18:33:50.117328 11052 raft_consensus.cc:2215] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c18cbbe25874ef99a282b8f1bd40dda in term 3. I0408 18:33:50.117533 11108 leader_election.cc:363] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 election: Vote granted by peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:50.117558 11108 leader_election.cc:258] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0408 18:33:50.117671 11376 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" candidate_term: 3 candidate_status { last_received { term: 1 index: 2502 } } ignore_live_leader: false dest_uuid: "b812540aa2e74153b4f96cc45fc3dd92" is_pre_election: true I0408 18:33:50.117715 11375 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" candidate_term: 3 candidate_status { last_received { term: 1 index: 2502 } } ignore_live_leader: false dest_uuid: "b812540aa2e74153b4f96cc45fc3dd92" I0408 18:33:50.117765 11376 raft_consensus.cc:2162] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c18cbbe25874ef99a282b8f1bd40dda for term 3 because replica has last-logged OpId of term: 1 index: 2503, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2502. I0408 18:33:50.117846 11375 raft_consensus.cc:2190] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c18cbbe25874ef99a282b8f1bd40dda for term 3 because replica is already servicing an update from a current leader or another vote. I0408 18:33:50.117911 11511 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda: Snoozing failure detection for 2.787s (election complete) I0408 18:33:50.117934 11511 raft_consensus.cc:2507] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 3 FOLLOWER]: Leader election won for term 3 I0408 18:33:50.117990 11511 raft_consensus.cc:572] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 3 LEADER]: Becoming Leader. State: Replica: 8c18cbbe25874ef99a282b8f1bd40dda, State: Running, Role: LEADER I0408 18:33:50.118039 11109 leader_election.cc:377] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 pre-election: Vote denied by peer b812540aa2e74153b4f96cc45fc3dd92. Message: Invalid argument: T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c18cbbe25874ef99a282b8f1bd40dda for term 3 because replica has last-logged OpId of term: 1 index: 2503, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2502. I0408 18:33:50.118072 11511 consensus_queue.cc:203] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2502, Committed index: 2502, Last appended: 1.2502, Last appended by leader: 2502, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:50.118175 11109 leader_election.cc:377] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [CANDIDATE]: Term 3 election: Vote denied by peer b812540aa2e74153b4f96cc45fc3dd92. Message: Service unavailable: T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c18cbbe25874ef99a282b8f1bd40dda for term 3 because replica is already servicing an update from a current leader or another vote. I0408 18:33:50.118891 10804 catalog_manager.cc:3595] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda reported cstate change: term changed from 1 to 3, leader changed from b812540aa2e74153b4f96cc45fc3dd92 (127.6.54.1) to 8c18cbbe25874ef99a282b8f1bd40dda (127.6.54.3). New cstate: current_term: 3 leader_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } health_report { overall_health: UNKNOWN } } } I0408 18:33:50.435554 11519 raft_consensus.cc:436] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0408 18:33:50.435637 11519 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Snoozing failure detection for 2.202s (starting election) I0408 18:33:50.435734 11519 raft_consensus.cc:458] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 } } peers { permanent_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 45705 } } peers { permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 } } I0408 18:33:50.435956 11519 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 312a97f4d00e453a8e99704e8073fa47 I0408 18:33:50.436015 11519 leader_election.cc:231] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c18cbbe25874ef99a282b8f1bd40dda I0408 18:33:50.439301 11169 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "b812540aa2e74153b4f96cc45fc3dd92" candidate_term: 2 candidate_status { last_received { term: 1 index: 2503 } } ignore_live_leader: false dest_uuid: "8c18cbbe25874ef99a282b8f1bd40dda" is_pre_election: true I0408 18:33:50.439316 11052 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ad5d88fd99744865a364185c5252c276" candidate_uuid: "b812540aa2e74153b4f96cc45fc3dd92" candidate_term: 2 candidate_status { last_received { term: 1 index: 2503 } } ignore_live_leader: false dest_uuid: "312a97f4d00e453a8e99704e8073fa47" is_pre_election: true I0408 18:33:50.439347 11169 raft_consensus.cc:2176] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate b812540aa2e74153b4f96cc45fc3dd92 for term 2 because replica is either leader or believes a valid leader to be alive. I0408 18:33:50.439347 11052 raft_consensus.cc:2120] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b812540aa2e74153b4f96cc45fc3dd92 for earlier term 2. Current term is 3. W0408 18:33:50.439604 11307 leader_election.cc:347] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 312a97f4d00e453a8e99704e8073fa47 with higher term. Message: Invalid argument: T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b812540aa2e74153b4f96cc45fc3dd92 for earlier term 2. Current term is 3. I0408 18:33:50.439626 11307 leader_election.cc:350] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term I0408 18:33:50.439690 11519 raft_consensus.cc:2700] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Snoozing failure detection for 1.619s (election complete) W0408 18:33:50.439690 11310 leader_election.cc:347] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c18cbbe25874ef99a282b8f1bd40dda with higher term. Message: Invalid argument: T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate b812540aa2e74153b4f96cc45fc3dd92 for term 2 because replica is either leader or believes a valid leader to be alive. I0408 18:33:50.439723 11519 raft_consensus.cc:2752] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 1 FOLLOWER]: Advancing to term 3 I0408 18:33:50.440332 11519 raft_consensus.cc:2455] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 312a97f4d00e453a8e99704e8073fa47 with higher term. Message: Invalid argument: T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b812540aa2e74153b4f96cc45fc3dd92 for earlier term 2. Current term is 3. I0408 18:33:50.565624 11052 raft_consensus.cc:1058] T ad5d88fd99744865a364185c5252c276 P 312a97f4d00e453a8e99704e8073fa47 [term 3 FOLLOWER]: Refusing update from remote peer 8c18cbbe25874ef99a282b8f1bd40dda: Log matching property violated. Preceding OpId in replica: term: 1 index: 2502. Preceding OpId from leader: term: 3 index: 2503. (index mismatch) I0408 18:33:50.565964 11511 consensus_queue.cc:962] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [LEADER]: Connected to new peer: Peer: permanent_uuid: "312a97f4d00e453a8e99704e8073fa47" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 44415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2503, Last known committed idx: 2501, Time since last communication: 0.000s W0408 18:33:50.566603 11527 rpcz_store.cc:251] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42546 (ReqId={client: a9ce32f4feb14299a4df4b824c22fad2, seq_no=2500, attempt_no=1}) took 3622ms (client timeout 1999). W0408 18:33:50.566721 11527 rpcz_store.cc:255] Trace: 0408 18:33:46.944397 (+ 0us) service_pool.cc:163] Inserting onto call queue 0408 18:33:46.944450 (+ 53us) service_pool.cc:222] Handling call 0408 18:33:50.566579 (+3622129us) inbound_call.cc:157] Queueing success response Metrics: {} I0408 18:33:50.567706 11375 raft_consensus.cc:1058] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92 [term 3 FOLLOWER]: Refusing update from remote peer 8c18cbbe25874ef99a282b8f1bd40dda: Log matching property violated. Preceding OpId in replica: term: 1 index: 2503. Preceding OpId from leader: term: 3 index: 2503. (term mismatch) I0408 18:33:50.567732 11375 pending_rounds.cc:78] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Aborting all transactions after (but not including) 2502 I0408 18:33:50.567742 11375 pending_rounds.cc:99] T ad5d88fd99744865a364185c5252c276 P b812540aa2e74153b4f96cc45fc3dd92: Aborting uncommitted WRITE_OP operation due to leader change: 1.2503 I0408 18:33:50.568125 11511 consensus_queue.cc:962] T ad5d88fd99744865a364185c5252c276 P 8c18cbbe25874ef99a282b8f1bd40dda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b812540aa2e74153b4f96cc45fc3dd92" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2503, Last known committed idx: 2502, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- bd60fa8277e54ef5899a976ca7608958 | 127.6.54.62:36789 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 312a97f4d00e453a8e99704e8073fa47 | 127.6.54.2:44415 | HEALTHY 8c18cbbe25874ef99a282b8f1bd40dda | 127.6.54.3:45705 | HEALTHY b812540aa2e74153b4f96cc45fc3dd92 | 127.6.54.1:34049 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.6.54.1 | experimental | 127.6.54.1:34049 local_ip_for_outbound_sockets | 127.6.54.2 | experimental | 127.6.54.2:44415 local_ip_for_outbound_sockets | 127.6.54.3 | experimental | 127.6.54.3:45705 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-0/data/info.pb | hidden | 127.6.54.1:34049 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-1/data/info.pb | hidden | 127.6.54.2:44415 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_0.1554748385377146-6360/minicluster-data/ts-2/data/info.pb | hidden | 127.6.54.3:45705 Version Summary Version | Servers --------------------------+------------------------- 1.7.0-cdh5.17.0-SNAPSHOT | all 4 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0408 18:33:50.598068 6360 log_verifier.cc:121] Checking tablet ad5d88fd99744865a364185c5252c276 I0408 18:33:50.717598 6360 log_verifier.cc:172] Verified matching terms for 2503 ops in tablet ad5d88fd99744865a364185c5252c276 I0408 18:33:50.718669 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 11297 W0408 18:33:50.721582 11108 connection.cc:513] server connection from 127.6.54.1:51911 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:50.721580 10987 connection.cc:513] server connection from 127.6.54.1:35189 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:50.721612 11109 connection.cc:513] client connection to 127.6.54.1:34049 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:50.721758 10787 connection.cc:513] server connection from 127.6.54.1:39843 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:50.721781 10778 connection.cc:513] client connection to 127.6.54.1:34049 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:50.721781 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 10974 W0408 18:33:50.724594 11108 connection.cc:513] client connection to 127.6.54.2:44415 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:50.724627 10790 connection.cc:513] server connection from 127.6.54.2:40763 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:50.724653 10787 connection.cc:513] client connection to 127.6.54.2:44415 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:50.724732 11108 connection.cc:513] server connection from 127.6.54.2:35989 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:50.724756 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 11098 W0408 18:33:50.727481 10790 connection.cc:513] client connection to 127.6.54.3:45705 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:50.727532 10789 connection.cc:513] server connection from 127.6.54.3:50627 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:50.727627 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 10779 W0408 18:33:50.730135 10778 connection.cc:513] client connection to 127.6.54.62:36789 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] DifferentFaultPoints/Kudu969Test.Test/0 (5806 ms) [ RUN ] DifferentFaultPoints/Kudu969Test.Test/1 I0408 18:33:50.738560 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:50.774133 11548 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:50.774286 11548 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:50.775014 11548 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0408 18:33:50.775029 11548 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:50.775033 11548 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0408 18:33:50.776113 11548 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.6.54.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb --webserver_interface=127.6.54.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.11548 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:50.777009 11548 master_main.cc:73] Initializing master server... I0408 18:33:50.777222 11548 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 34011us I0408 18:33:50.777386 11548 fs_manager.cc:260] Metadata directory not provided I0408 18:33:50.777413 11548 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:50.777472 11548 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0408 18:33:50.777487 11548 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:50.778797 11548 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/instance: uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.778945 11548 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/wal/instance: uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.780200 11548 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:50.780424 11548 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:50.780450 11548 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:50.781029 11548 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:50.781059 11548 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/wal uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.781102 11548 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:50.800531 11548 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:50.800609 11548 master_main.cc:76] Starting Master server... I0408 18:33:50.800745 11563 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:50.800767 11563 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:50.800773 11563 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:50.802801 11548 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.62:46549 I0408 18:33:50.802861 11548 webserver.cc:173] Starting webserver on 127.6.54.62:0 I0408 18:33:50.802893 11548 webserver.cc:184] Document root disabled I0408 18:33:50.803103 11548 webserver.cc:311] Webserver started. Bound to: http://127.6.54.62:45653/ I0408 18:33:50.803568 11548 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb I0408 18:33:50.804014 11607 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:50.806162 11607 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b: Bootstrap starting. I0408 18:33:50.806627 11607 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b: No blocks or log segments found. Creating new log. I0408 18:33:50.806859 11607 log.cc:520] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b: Log is configured to *not* fsync() on all Append() calls I0408 18:33:50.807310 11607 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b: No bootstrap required, opened a new log I0408 18:33:50.808643 11607 raft_consensus.cc:304] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" member_type: VOTER } I0408 18:33:50.808681 11607 raft_consensus.cc:330] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:50.808696 11607 raft_consensus.cc:605] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c5e6b7d8a34af28738dbbe1d654b2b, State: Initialized, Role: FOLLOWER I0408 18:33:50.808817 11607 consensus_queue.cc:226] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" member_type: VOTER } I0408 18:33:50.808837 11607 raft_consensus.cc:344] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0408 18:33:50.808849 11607 raft_consensus.cc:436] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0408 18:33:50.808885 11607 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b: Snoozing failure detection for 1.506s (starting election) I0408 18:33:50.808910 11607 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:50.809337 11607 raft_consensus.cc:458] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" member_type: VOTER } I0408 18:33:50.809399 11607 leader_election.cc:258] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:50.809567 11609 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b: Snoozing failure detection for 1.898s (election complete) I0408 18:33:50.809592 11609 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:50.809666 11609 raft_consensus.cc:572] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [term 1 LEADER]: Becoming Leader. State: Replica: 74c5e6b7d8a34af28738dbbe1d654b2b, State: Running, Role: LEADER I0408 18:33:50.809736 11609 consensus_queue.cc:203] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" member_type: VOTER } I0408 18:33:50.809939 11607 sys_catalog.cc:335] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" member_type: VOTER } } I0408 18:33:50.809939 11609 sys_catalog.cc:335] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" member_type: VOTER } } I0408 18:33:50.809979 11607 sys_catalog.cc:338] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [sys.catalog]: This master's current role is: LEADER I0408 18:33:50.809995 11610 sys_catalog.cc:335] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c5e6b7d8a34af28738dbbe1d654b2b. Latest consensus state: current_term: 1 leader_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5e6b7d8a34af28738dbbe1d654b2b" member_type: VOTER } } I0408 18:33:50.810111 11610 sys_catalog.cc:338] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [sys.catalog]: This master's current role is: LEADER I0408 18:33:50.810026 11609 sys_catalog.cc:338] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [sys.catalog]: This master's current role is: LEADER I0408 18:33:50.810201 11607 sys_catalog.cc:422] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b [sys.catalog]: configured and running, proceeding with master startup. I0408 18:33:50.810497 11548 master_main.cc:79] Master server successfully started. W0408 18:33:50.810721 11614 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0408 18:33:50.810739 11614 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0408 18:33:50.810782 11613 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0408 18:33:50.810904 11613 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0408 18:33:50.812265 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master as pid 11548 I0408 18:33:50.812496 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:0 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --maintenance_manager_num_threads=3 with env {} I0408 18:33:50.831454 11613 catalog_manager.cc:870] Generated new certificate authority record I0408 18:33:50.832186 11613 catalog_manager.cc:992] Loading token signing keys... I0408 18:33:50.840987 11613 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 74c5e6b7d8a34af28738dbbe1d654b2b: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:50.872099 11616 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:50.872318 11616 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:50.873322 11616 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:50.873356 11616 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:50.874794 11616 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=0 --tserver_master_addrs=127.6.54.62:46549 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11616 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --maintenance_manager_num_threads=3 --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:50.876338 11616 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:50.876672 11616 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 34011us I0408 18:33:50.876857 11616 fs_manager.cc:260] Metadata directory not provided I0408 18:33:50.876896 11616 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:50.876987 11616 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0408 18:33:50.877012 11616 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:50.878947 11616 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/instance: uuid: "3502ea6d66a64ab2b1aaed18d282ebec" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.879187 11616 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/wal/instance: uuid: "3502ea6d66a64ab2b1aaed18d282ebec" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.887495 11616 fs_manager.cc:495] Time spent creating directory manager: real 0.008s user 0.002s sys 0.000s I0408 18:33:50.887895 11616 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:50.887940 11616 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:50.888800 11616 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:50.888849 11616 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "3502ea6d66a64ab2b1aaed18d282ebec" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.888911 11616 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:50.909772 11616 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:50.910024 11616 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:50.910115 11616 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:50.914844 11616 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:43137 I0408 18:33:50.914927 11616 webserver.cc:173] Starting webserver on 127.6.54.1:0 I0408 18:33:50.914948 11616 webserver.cc:184] Document root disabled I0408 18:33:50.916293 11616 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:41447/ I0408 18:33:50.916615 11616 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:50.917758 11616 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:50.919409 11740 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:50.919425 11740 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:50.919430 11740 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:50.924412 11739 heartbeater.cc:346] Connected to a master server at 127.6.54.62:46549 I0408 18:33:50.924458 11739 heartbeater.cc:426] Registering TS with master... I0408 18:33:50.924562 11739 heartbeater.cc:475] Master 127.6.54.62:46549 requested a full tablet report, sending... I0408 18:33:50.925036 11568 ts_manager.cc:81] Registered new tserver with Master: 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1:43137) I0408 18:33:50.925112 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 11616 I0408 18:33:50.925415 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.6.54.2:0 --local_ip_for_outbound_sockets=127.6.54.2 --webserver_interface=127.6.54.2 --webserver_port=0 --tserver_master_addrs=127.6.54.62:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --maintenance_manager_num_threads=3 with env {} I0408 18:33:50.925900 11568 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:43795 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:50.959694 11743 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:50.959836 11743 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:50.960551 11743 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:50.960577 11743 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.2 I0408 18:33:50.961539 11743 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/info.pb --webserver_interface=127.6.54.2 --webserver_port=0 --tserver_master_addrs=127.6.54.62:46549 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11743 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --maintenance_manager_num_threads=3 --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.2 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:50.962604 11743 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:50.962847 11743 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 34011us I0408 18:33:50.962994 11743 fs_manager.cc:260] Metadata directory not provided I0408 18:33:50.963028 11743 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:50.963109 11743 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0408 18:33:50.963127 11743 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:50.964509 11743 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/instance: uuid: "6c9d88f121474dd99a68c2b254085723" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.964705 11743 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/wal/instance: uuid: "6c9d88f121474dd99a68c2b254085723" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.965955 11743 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0408 18:33:50.966289 11743 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:50.966317 11743 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:50.967170 11743 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:50.967202 11743 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/wal uuid: "6c9d88f121474dd99a68c2b254085723" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:50.967267 11743 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:51.001399 11743 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:51.001729 11743 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:51.001808 11743 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:51.006201 11743 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.2:34299 I0408 18:33:51.006286 11743 webserver.cc:173] Starting webserver on 127.6.54.2:0 I0408 18:33:51.006302 11743 webserver.cc:184] Document root disabled I0408 18:33:51.006510 11743 webserver.cc:311] Webserver started. Bound to: http://127.6.54.2:38437/ I0408 18:33:51.006743 11743 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/info.pb I0408 18:33:51.007299 11743 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:51.007194 11865 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:51.007333 11865 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:51.007339 11865 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:51.010197 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 11743 I0408 18:33:51.010509 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.6.54.3:0 --local_ip_for_outbound_sockets=127.6.54.3 --webserver_interface=127.6.54.3 --webserver_port=0 --tserver_master_addrs=127.6.54.62:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --maintenance_manager_num_threads=3 with env {} I0408 18:33:51.011014 11864 heartbeater.cc:346] Connected to a master server at 127.6.54.62:46549 I0408 18:33:51.011045 11864 heartbeater.cc:426] Registering TS with master... I0408 18:33:51.011128 11864 heartbeater.cc:475] Master 127.6.54.62:46549 requested a full tablet report, sending... I0408 18:33:51.011545 11568 ts_manager.cc:81] Registered new tserver with Master: 6c9d88f121474dd99a68c2b254085723 (127.6.54.2:34299) I0408 18:33:51.012218 11568 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.2:38745 WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:51.046581 11867 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:51.046723 11867 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:51.047734 11867 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:51.047762 11867 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.3 I0408 18:33:51.048851 11867 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/info.pb --webserver_interface=127.6.54.3 --webserver_port=0 --tserver_master_addrs=127.6.54.62:46549 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11867 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --maintenance_manager_num_threads=3 --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.3 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:51.049923 11867 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:51.050256 11867 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 34511us I0408 18:33:51.050388 11867 fs_manager.cc:260] Metadata directory not provided I0408 18:33:51.050415 11867 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:51.050487 11867 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0408 18:33:51.050505 11867 server_base.cc:435] Attempting to create new FS layout instead I0408 18:33:51.051954 11867 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/instance: uuid: "87aa5becfacc4cf39bdcb08f191990b8" format_stamp: "Formatted at 2019-04-08 18:33:51 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:51.052129 11867 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/wal/instance: uuid: "87aa5becfacc4cf39bdcb08f191990b8" format_stamp: "Formatted at 2019-04-08 18:33:51 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:51.053427 11867 fs_manager.cc:495] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0408 18:33:51.053722 11867 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:51.053751 11867 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:51.054517 11867 fs_manager.cc:417] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0408 18:33:51.054559 11867 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/wal uuid: "87aa5becfacc4cf39bdcb08f191990b8" format_stamp: "Formatted at 2019-04-08 18:33:51 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:51.054610 11867 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:51.070859 11867 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:51.071066 11867 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0408 18:33:51.071131 11867 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:51.075487 11867 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.3:44449 I0408 18:33:51.075588 11867 webserver.cc:173] Starting webserver on 127.6.54.3:0 I0408 18:33:51.075605 11867 webserver.cc:184] Document root disabled I0408 18:33:51.075830 11867 webserver.cc:311] Webserver started. Bound to: http://127.6.54.3:37425/ I0408 18:33:51.076063 11867 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/info.pb I0408 18:33:51.076406 11867 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:51.076468 11989 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:51.076486 11989 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:51.076491 11989 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:51.080148 11988 heartbeater.cc:346] Connected to a master server at 127.6.54.62:46549 I0408 18:33:51.080212 11988 heartbeater.cc:426] Registering TS with master... I0408 18:33:51.080307 11988 heartbeater.cc:475] Master 127.6.54.62:46549 requested a full tablet report, sending... I0408 18:33:51.080643 11568 ts_manager.cc:81] Registered new tserver with Master: 87aa5becfacc4cf39bdcb08f191990b8 (127.6.54.3:44449) I0408 18:33:51.081142 11568 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.3:49031 I0408 18:33:51.084812 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 11867 I0408 18:33:51.087421 6360 external_mini_cluster.cc:499] 3 TS(s) registered with all masters W0408 18:33:51.094708 11736 generic_service.cc:126] {username='slave'} at 127.0.0.1:42360 forcing change of non-runtime-safe flag flush_threshold_mb I0408 18:33:51.094741 11736 generic_service.cc:146] {username='slave'} at 127.0.0.1:42360 changed flags via RPC: flush_threshold_mb from '1024' to '1' I0408 18:33:51.094982 6360 test_util.cc:200] Using random seed: 334714054 I0408 18:33:51.099457 11568 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36494: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0408 18:33:51.099648 11568 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0408 18:33:51.104348 11673 tablet_service.cc:758] Processing CreateTablet for tablet dd1b98d584d5486294a0219bf1a80f1b (table=test-workload [id=64c520a98bcc4727b52b47ab1b98e5a5]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:51.104540 11673 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet dd1b98d584d5486294a0219bf1a80f1b. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:51.105432 11925 tablet_service.cc:758] Processing CreateTablet for tablet dd1b98d584d5486294a0219bf1a80f1b (table=test-workload [id=64c520a98bcc4727b52b47ab1b98e5a5]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:51.105595 11925 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet dd1b98d584d5486294a0219bf1a80f1b. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:51.105609 11801 tablet_service.cc:758] Processing CreateTablet for tablet dd1b98d584d5486294a0219bf1a80f1b (table=test-workload [id=64c520a98bcc4727b52b47ab1b98e5a5]), partition=RANGE (key) PARTITION UNBOUNDED I0408 18:33:51.105736 11801 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet dd1b98d584d5486294a0219bf1a80f1b. 1 dirs total, 0 dirs full, 0 dirs failed I0408 18:33:51.106446 11673 ts_tablet_manager.cc:1080] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:51.107947 11925 ts_tablet_manager.cc:1080] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:51.108027 11801 ts_tablet_manager.cc:1080] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:51.108186 12008 ts_tablet_manager.cc:938] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Bootstrapping tablet I0408 18:33:51.108271 12008 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Bootstrap starting. I0408 18:33:51.108686 12008 tablet_bootstrap.cc:581] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: No blocks or log segments found. Creating new log. I0408 18:33:51.108897 12008 log.cc:520] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Log is configured to *not* fsync() on all Append() calls I0408 18:33:51.109019 12009 ts_tablet_manager.cc:938] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Bootstrapping tablet I0408 18:33:51.109069 12009 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Bootstrap starting. I0408 18:33:51.109364 12008 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: No bootstrap required, opened a new log I0408 18:33:51.109400 12009 tablet_bootstrap.cc:581] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: No blocks or log segments found. Creating new log. I0408 18:33:51.109413 12008 ts_tablet_manager.cc:955] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:51.109617 12009 log.cc:520] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Log is configured to *not* fsync() on all Append() calls I0408 18:33:51.109755 12010 ts_tablet_manager.cc:938] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Bootstrapping tablet I0408 18:33:51.109856 12010 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Bootstrap starting. I0408 18:33:51.110036 12009 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: No bootstrap required, opened a new log I0408 18:33:51.110139 12009 ts_tablet_manager.cc:955] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0408 18:33:51.110278 12010 tablet_bootstrap.cc:581] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: No blocks or log segments found. Creating new log. I0408 18:33:51.110500 12010 log.cc:520] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Log is configured to *not* fsync() on all Append() calls I0408 18:33:51.110906 12008 raft_consensus.cc:304] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.110957 12008 raft_consensus.cc:330] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:51.110980 12008 raft_consensus.cc:605] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3502ea6d66a64ab2b1aaed18d282ebec, State: Initialized, Role: FOLLOWER I0408 18:33:51.111111 12008 consensus_queue.cc:226] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.111310 11739 heartbeater.cc:467] Master 127.6.54.62:46549 was elected leader, sending a full tablet report... I0408 18:33:51.111336 12008 ts_tablet_manager.cc:983] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:51.111594 12010 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: No bootstrap required, opened a new log I0408 18:33:51.111649 12010 ts_tablet_manager.cc:955] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0408 18:33:51.111778 12009 raft_consensus.cc:304] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.111815 12009 raft_consensus.cc:330] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:51.111835 12009 raft_consensus.cc:605] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87aa5becfacc4cf39bdcb08f191990b8, State: Initialized, Role: FOLLOWER I0408 18:33:51.111966 12009 consensus_queue.cc:226] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.112118 12009 ts_tablet_manager.cc:983] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:51.112124 11988 heartbeater.cc:467] Master 127.6.54.62:46549 was elected leader, sending a full tablet report... I0408 18:33:51.113003 12010 raft_consensus.cc:304] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.113039 12010 raft_consensus.cc:330] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0408 18:33:51.113050 12010 raft_consensus.cc:605] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9d88f121474dd99a68c2b254085723, State: Initialized, Role: FOLLOWER I0408 18:33:51.113154 12010 consensus_queue.cc:226] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.113302 11864 heartbeater.cc:467] Master 127.6.54.62:46549 was elected leader, sending a full tablet report... I0408 18:33:51.113312 12010 ts_tablet_manager.cc:983] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0408 18:33:51.116152 12014 raft_consensus.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0408 18:33:51.116194 12014 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Snoozing failure detection for 1.620s (starting election) I0408 18:33:51.116245 12014 raft_consensus.cc:458] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.116468 12014 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 87aa5becfacc4cf39bdcb08f191990b8 I0408 18:33:51.116540 12014 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6c9d88f121474dd99a68c2b254085723 I0408 18:33:51.119210 11945 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dd1b98d584d5486294a0219bf1a80f1b" candidate_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87aa5becfacc4cf39bdcb08f191990b8" is_pre_election: true I0408 18:33:51.119284 11945 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Snoozing failure detection for 1.798s (vote granted) I0408 18:33:51.119313 11945 raft_consensus.cc:2215] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3502ea6d66a64ab2b1aaed18d282ebec in term 0. I0408 18:33:51.119442 11821 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dd1b98d584d5486294a0219bf1a80f1b" candidate_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9d88f121474dd99a68c2b254085723" is_pre_election: true I0408 18:33:51.119500 11821 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Snoozing failure detection for 1.913s (vote granted) I0408 18:33:51.119518 11821 raft_consensus.cc:2215] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3502ea6d66a64ab2b1aaed18d282ebec in term 0. I0408 18:33:51.119573 11630 leader_election.cc:363] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 pre-election: Vote granted by peer 87aa5becfacc4cf39bdcb08f191990b8 I0408 18:33:51.119591 11630 leader_election.cc:258] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0408 18:33:51.119644 12014 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Snoozing failure detection for 1.768s (election complete) I0408 18:33:51.119678 12014 raft_consensus.cc:2507] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 0 FOLLOWER]: Leader pre-election won for term 1 I0408 18:33:51.119709 12014 raft_consensus.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0408 18:33:51.119719 12014 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Snoozing failure detection for 2.179s (starting election) I0408 18:33:51.119719 11631 leader_election.cc:363] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6c9d88f121474dd99a68c2b254085723 I0408 18:33:51.119726 12014 raft_consensus.cc:2752] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:51.120282 12014 raft_consensus.cc:458] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.120390 12014 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 election: Requesting vote from peer 87aa5becfacc4cf39bdcb08f191990b8 I0408 18:33:51.120419 12014 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 election: Requesting vote from peer 6c9d88f121474dd99a68c2b254085723 I0408 18:33:51.120673 11945 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dd1b98d584d5486294a0219bf1a80f1b" candidate_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87aa5becfacc4cf39bdcb08f191990b8" I0408 18:33:51.120676 11821 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dd1b98d584d5486294a0219bf1a80f1b" candidate_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9d88f121474dd99a68c2b254085723" I0408 18:33:51.120704 11945 raft_consensus.cc:2752] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:51.120704 11821 raft_consensus.cc:2752] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 0 FOLLOWER]: Advancing to term 1 I0408 18:33:51.120740 11821 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Snoozing failure detection for 2.039s (vote granted) I0408 18:33:51.120740 11945 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Snoozing failure detection for 1.694s (vote granted) I0408 18:33:51.121366 11821 raft_consensus.cc:2215] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3502ea6d66a64ab2b1aaed18d282ebec in term 1. I0408 18:33:51.121366 11945 raft_consensus.cc:2215] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3502ea6d66a64ab2b1aaed18d282ebec in term 1. I0408 18:33:51.121562 11631 leader_election.cc:363] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 election: Vote granted by peer 6c9d88f121474dd99a68c2b254085723 I0408 18:33:51.121626 11631 leader_election.cc:258] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0408 18:33:51.121659 11630 leader_election.cc:363] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [CANDIDATE]: Term 1 election: Vote granted by peer 87aa5becfacc4cf39bdcb08f191990b8 I0408 18:33:51.121697 12014 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Snoozing failure detection for 1.699s (election complete) I0408 18:33:51.121718 12014 raft_consensus.cc:2507] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 1 FOLLOWER]: Leader election won for term 1 I0408 18:33:51.121796 12014 raft_consensus.cc:572] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 1 LEADER]: Becoming Leader. State: Replica: 3502ea6d66a64ab2b1aaed18d282ebec, State: Running, Role: LEADER I0408 18:33:51.121877 12014 consensus_queue.cc:203] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:51.122733 11565 catalog_manager.cc:3595] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec reported cstate change: term changed from 0 to 1, leader changed from to 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1). New cstate: current_term: 1 leader_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } health_report { overall_health: UNKNOWN } } } W0408 18:33:51.134461 11736 generic_service.cc:126] {username='slave'} at 127.0.0.1:42360 forcing change of non-runtime-safe flag fault_crash_before_flush_tablet_meta_after_flush_mrs I0408 18:33:51.134555 11736 generic_service.cc:146] {username='slave'} at 127.0.0.1:42360 changed flags via RPC: fault_crash_before_flush_tablet_meta_after_flush_mrs from '0' to '0.3' I0408 18:33:51.152089 11821 raft_consensus.cc:1058] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 1 FOLLOWER]: Refusing update from remote peer 3502ea6d66a64ab2b1aaed18d282ebec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0408 18:33:51.152103 11945 raft_consensus.cc:1058] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 1 FOLLOWER]: Refusing update from remote peer 3502ea6d66a64ab2b1aaed18d282ebec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0408 18:33:51.152484 12018 consensus_queue.cc:962] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0408 18:33:51.152576 12014 consensus_queue.cc:962] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [LEADER]: Connected to new peer: Peer: permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0408 18:33:51.669968 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.945090 I0408 18:33:51.670239 11625 tablet.cc:1153] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:51.670258 11625 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:51.670275 11625 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:51.670281 11625 tablet.cc:1155] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Memstore in-memory size: 2039575 bytes I0408 18:33:51.670310 11625 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249576113102848 or (T in {6368249576113102848})}] I0408 18:33:51.671664 11625 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:51.681404 11625 tablet.cc:1529] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:51.681555 11625 tablet.cc:1584] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:51.681632 11625 tablet.cc:1586] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249576158367744 or (T in {6368249576158367744})}] I0408 18:33:51.682879 11625 delta_tracker.cc:722] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushing 15 deltas (1759 bytes in memory) from DMS 0 I0408 18:33:51.683185 11625 delta_tracker.cc:663] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushed delta block 0000000000000008 (420 bytes on disk) stats: ts range=[6368249576120303616, 6368249576160415744], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:15] W0408 18:33:51.683284 11625 fault_injection.cc:41] FAULT INJECTION ENABLED! W0408 18:33:51.683327 11625 fault_injection.cc:42] THIS SERVER MAY CRASH! I0408 18:33:51.684234 11625 tablet.cc:1634] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush successful on 10600 rows (52795 bytes) I0408 18:33:51.684392 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"cfile_init":1,"compiler_manager_pool.queue_time_us":2465,"data dirs.queue_time_us":705,"data dirs.run_cpu_time_us":109,"data dirs.run_wall_time_us":2537,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":38,"thread_start_us":424,"threads_started":2} I0408 18:33:51.684609 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 98531 bytes on disk I0408 18:33:51.684660 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 98531 bytes on disk I0408 18:33:51.684687 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 98531 bytes on disk I0408 18:33:51.684830 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4} I0408 18:33:51.684870 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:51.684891 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:52.435382 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.945090 I0408 18:33:52.435559 11625 tablet.cc:1153] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:52.435573 11625 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:52.435590 11625 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:52.435595 11625 tablet.cc:1155] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Memstore in-memory size: 2039575 bytes I0408 18:33:52.435606 11625 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249579249221632 or (T in {6368249579249221632})}] I0408 18:33:52.435777 11625 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:52.441423 11625 tablet.cc:1529] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:52.441462 11625 tablet.cc:1584] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:52.441469 11625 tablet.cc:1586] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249579249221632 or (T in {6368249579249221632})}] I0408 18:33:52.443512 11625 tablet.cc:1634] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush successful on 13900 rows (58752 bytes) I0408 18:33:52.443673 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.008s user 0.007s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":84,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":720,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":340,"lbm_writes_lt_1ms":37,"thread_start_us":43,"threads_started":1} I0408 18:33:52.443833 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 119178 bytes on disk I0408 18:33:52.443893 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.989939 I0408 18:33:52.443940 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.989939 I0408 18:33:52.443974 11623 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4} I0408 18:33:52.443992 11624 tablet.cc:1664] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:52.444000 11624 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:52.444007 11624 compaction.cc:917] RowSet(0)(current size on disk: ~151746 bytes) I0408 18:33:52.444013 11624 compaction.cc:917] RowSet(1)(current size on disk: ~177930 bytes) I0408 18:33:52.444017 11623 tablet.cc:1664] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: stage 1 complete, picked 0 rowsets to compact or flush I0408 18:33:52.444020 11624 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249579285843968}] I0408 18:33:52.444025 11623 compaction.cc:914] Selected 0 rowsets to compact: I0408 18:33:52.444030 11623 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249579285843968}] I0408 18:33:52.444114 11623 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:52.444386 11624 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:52.444772 11623 tablet.cc:1473] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets. I0408 18:33:52.446017 11623 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"data dirs.queue_time_us":954,"lbm_write_time_us":58,"lbm_writes_lt_1ms":6,"spinlock_wait_cycles":3200} I0408 18:33:52.466372 11624 tablet.cc:1529] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:52.466468 11624 tablet.cc:1584] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:52.466513 11624 tablet.cc:1586] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249579374092288 or (T in {6368249579374092288})}] I0408 18:33:52.470691 11624 delta_tracker.cc:722] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushing 37 deltas (3751 bytes in memory) from DMS 0 I0408 18:33:52.470935 11624 delta_tracker.cc:663] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushed delta block 0000000000000027 (803 bytes on disk) stats: ts range=[6368249579285843968, 6368249579390595072], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:37] I0408 18:33:52.472015 11624 tablet.cc:1634] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction successful on 24500 rows (115238 bytes) I0408 18:33:52.472491 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.028s user 0.022s sys 0.001s Metrics: {"cfile_cache_hit":15,"cfile_cache_hit_bytes":10281,"cfile_cache_miss":47,"cfile_cache_miss_bytes":515443,"cfile_init":5,"data dirs.queue_time_us":90,"data dirs.run_cpu_time_us":261,"data dirs.run_wall_time_us":2558,"delta_iterators_relevant":6,"lbm_read_time_us":433,"lbm_reads_lt_1ms":67,"lbm_write_time_us":806,"lbm_writes_lt_1ms":53} I0408 18:33:52.472662 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 228364 bytes on disk I0408 18:33:52.472705 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 228364 bytes on disk I0408 18:33:52.472733 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 228364 bytes on disk I0408 18:33:52.472842 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4} I0408 18:33:52.472885 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:52.472923 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:53.223314 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.945090 I0408 18:33:53.223475 11624 tablet.cc:1153] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:53.223491 11624 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:53.223497 11624 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:53.223502 11624 tablet.cc:1155] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Memstore in-memory size: 2039575 bytes I0408 18:33:53.223511 11624 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249582476701696 or (T in {6368249582476701696})}] I0408 18:33:53.223659 11624 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:53.229419 11624 tablet.cc:1529] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:53.229452 11624 tablet.cc:1584] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:53.229459 11624 tablet.cc:1586] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249582481510400 or (T in {6368249582481510400})}] I0408 18:33:53.230042 11624 delta_tracker.cc:722] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushing 2 deltas (763 bytes in memory) from DMS 0 I0408 18:33:53.230156 11624 delta_tracker.cc:663] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushed delta block 0000000000000034 (196 bytes on disk) stats: ts range=[6368249582477201408, 6368249582481510400], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:2] I0408 18:33:53.231623 11624 tablet.cc:1634] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush successful on 13800 rows (53537 bytes) I0408 18:33:53.231781 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.008s user 0.004s sys 0.003s Metrics: {"cfile_init":1,"data dirs.queue_time_us":98,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":1090,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":320,"lbm_writes_lt_1ms":41,"thread_start_us":53,"threads_started":1} I0408 18:33:53.231933 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.989890 I0408 18:33:53.231987 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 115814 bytes on disk I0408 18:33:53.232017 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 115814 bytes on disk I0408 18:33:53.232059 11623 tablet.cc:1664] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:53.232067 11623 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:53.232072 11623 compaction.cc:917] RowSet(3)(current size on disk: ~344405 bytes) I0408 18:33:53.232077 11623 compaction.cc:917] RowSet(4)(current size on disk: ~169547 bytes) I0408 18:33:53.232084 11623 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249582512312320}] I0408 18:33:53.232203 11623 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:53.235204 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:53.235236 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:53.235286 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.989890 I0408 18:33:53.235359 11625 tablet.cc:1664] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: stage 1 complete, picked 0 rowsets to compact or flush I0408 18:33:53.235368 11625 compaction.cc:914] Selected 0 rowsets to compact: I0408 18:33:53.235376 11625 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249582512312320}] I0408 18:33:53.235472 11625 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:53.235971 11625 tablet.cc:1473] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets. I0408 18:33:53.239734 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.004s user 0.001s sys 0.000s Metrics: {"data dirs.queue_time_us":10901,"data dirs.run_cpu_time_us":270,"data dirs.run_wall_time_us":3431,"lbm_write_time_us":75,"lbm_writes_lt_1ms":6} I0408 18:33:53.257725 11623 tablet.cc:1529] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:53.257850 11623 tablet.cc:1584] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:53.257874 11623 tablet.cc:1586] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249582612680704 or (T in {6368249582612680704})}] I0408 18:33:53.264516 11623 delta_tracker.cc:722] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushing 31 deltas (3751 bytes in memory) from DMS 0 I0408 18:33:53.264791 11623 delta_tracker.cc:663] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushed delta block 0000000000000047 (705 bytes on disk) stats: ts range=[6368249582512312320, 6368249582629330944], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:31] I0408 18:33:53.265774 11623 tablet.cc:1634] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction successful on 38300 rows (178594 bytes) I0408 18:33:53.266058 11623 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.034s user 0.032s sys 0.000s Metrics: {"cfile_cache_hit":15,"cfile_cache_hit_bytes":11833,"cfile_cache_miss":65,"cfile_cache_miss_bytes":816315,"cfile_init":7,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":175,"data dirs.run_wall_time_us":719,"delta_iterators_relevant":8,"lbm_read_time_us":636,"lbm_reads_lt_1ms":93,"lbm_write_time_us":1075,"lbm_writes_lt_1ms":68} I0408 18:33:53.266212 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 357798 bytes on disk I0408 18:33:53.266255 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 357798 bytes on disk I0408 18:33:53.266294 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 357798 bytes on disk I0408 18:33:53.266376 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4} I0408 18:33:53.266415 11623 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:53.266420 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:53.766754 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.945090 I0408 18:33:53.766918 11625 tablet.cc:1153] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:53.766934 11625 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:53.766940 11625 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:53.766945 11625 tablet.cc:1155] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Memstore in-memory size: 2039575 bytes I0408 18:33:53.766952 11625 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249584706117632}] I0408 18:33:53.767094 11625 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:53.774703 11625 tablet.cc:1529] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:53.774901 11625 tablet.cc:1584] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:53.774958 11625 tablet.cc:1586] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249584734928896 or (T in {6368249584734928896})}] I0408 18:33:53.775851 11625 delta_tracker.cc:722] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushing 2 deltas (763 bytes in memory) from DMS 0 I0408 18:33:53.776029 11625 delta_tracker.cc:663] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushed delta block 0000000000000054 (198 bytes on disk) stats: ts range=[6368249584711483392, 6368249584723546112], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:2] I0408 18:33:53.777058 11625 tablet.cc:1634] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush successful on 10400 rows (38741 bytes) I0408 18:33:53.777191 11625 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.010s user 0.003s sys 0.005s Metrics: {"cfile_init":1,"data dirs.queue_time_us":93,"data dirs.run_cpu_time_us":90,"data dirs.run_wall_time_us":630,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":725,"lbm_writes_lt_1ms":37,"spinlock_wait_cycles":1920,"thread_start_us":63,"threads_started":1} I0408 18:33:53.777582 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.989823 I0408 18:33:53.777638 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 86106 bytes on disk I0408 18:33:53.777671 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.989823 I0408 18:33:53.777714 11623 tablet.cc:1664] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0408 18:33:53.777720 11623 compaction.cc:914] Selected 2 rowsets to compact: I0408 18:33:53.777725 11623 compaction.cc:917] RowSet(5)(current size on disk: ~537097 bytes) I0408 18:33:53.777729 11623 compaction.cc:917] RowSet(7)(current size on disk: ~125045 bytes) I0408 18:33:53.777737 11623 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249584746250240 or (T in {6368249584746250240})}] I0408 18:33:53.777828 11623 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:53.779273 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:53.779299 11624 tablet.cc:1664] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: stage 1 complete, picked 0 rowsets to compact or flush I0408 18:33:53.779304 11624 compaction.cc:914] Selected 0 rowsets to compact: I0408 18:33:53.779310 11624 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249584746250240 or (T in {6368249584746250240})}] I0408 18:33:53.779381 11624 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:53.779578 11624 tablet.cc:1473] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets. I0408 18:33:53.780217 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"data dirs.queue_time_us":34,"lbm_write_time_us":48,"lbm_writes_lt_1ms":6} I0408 18:33:53.810109 11623 tablet.cc:1529] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:53.810237 11623 tablet.cc:1584] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:53.810297 11623 tablet.cc:1586] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249584869642240 or (T in {6368249584869642240})}] I0408 18:33:53.818683 11623 delta_tracker.cc:722] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushing 38 deltas (3751 bytes in memory) from DMS 0 I0408 18:33:53.818924 11623 delta_tracker.cc:663] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushed delta block 0000000000000067 (821 bytes on disk) stats: ts range=[6368249584762892288, 6368249584909950976], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:38] I0408 18:33:53.822299 11623 tablet.cc:1634] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Compaction successful on 48700 rows (224318 bytes) I0408 18:33:53.822381 11623 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: CompactRowSetsOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.045s user 0.035s sys 0.004s Metrics: {"cfile_cache_hit":16,"cfile_cache_hit_bytes":18207,"cfile_cache_miss":74,"cfile_cache_miss_bytes":1056885,"cfile_init":6,"data dirs.queue_time_us":19,"data dirs.run_cpu_time_us":79,"data dirs.run_wall_time_us":632,"delta_iterators_relevant":8,"lbm_read_time_us":739,"lbm_reads_lt_1ms":98,"lbm_write_time_us":1551,"lbm_writes_lt_1ms":79} I0408 18:33:53.822469 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 453782 bytes on disk I0408 18:33:53.822538 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 453782 bytes on disk I0408 18:33:53.822568 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 453782 bytes on disk I0408 18:33:53.822656 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0408 18:33:53.822695 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:53.822716 11624 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0408 18:33:54.573225 11740 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling FlushMRSOp(dd1b98d584d5486294a0219bf1a80f1b): perf score=0.945090 I0408 18:33:54.574291 11623 tablet.cc:1153] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering stage 1 (old memrowset already frozen for inserts) I0408 18:33:54.574308 11623 compaction.cc:914] Selected 1 rowsets to compact: I0408 18:33:54.574314 11623 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0408 18:33:54.574319 11623 tablet.cc:1155] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Memstore in-memory size: 2039575 bytes I0408 18:33:54.574328 11623 tablet.cc:1444] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6368249588007067648}] I0408 18:33:54.574460 11623 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0408 18:33:54.583146 11623 tablet.cc:1529] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0408 18:33:54.583259 11623 tablet.cc:1584] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0408 18:33:54.583318 11623 tablet.cc:1586] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6368249588042129408 or (T in {6368249588042129408})}] I0408 18:33:54.584367 11623 delta_tracker.cc:722] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushing 2 deltas (763 bytes in memory) from DMS 0 I0408 18:33:54.584597 11623 delta_tracker.cc:663] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Flushed delta block 0000000000000074 (198 bytes on disk) stats: ts range=[6368249588017840128, 6368249588018016256], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[1:2] E0408 18:33:54.584636 11623 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_flush_tablet_meta_after_flush_mrs (process will exit) W0408 18:33:54.589920 12021 connection.cc:513] client connection to 127.6.54.1:43137 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:54.589915 11753 connection.cc:513] server connection from 127.6.54.1:56747 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:54.589933 11877 connection.cc:513] server connection from 127.6.54.1:46855 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:54.590088 11877 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0408 18:33:54.590090 11753 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0408 18:33:54.590099 12021 meta_cache.cc:219] Tablet dd1b98d584d5486294a0219bf1a80f1b: Replica 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1:43137) has failed: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:54.590103 11547 connection.cc:513] client connection to 127.6.54.1:43137 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:54.590229 12021 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0408 18:33:54.590304 11557 connection.cc:513] client connection to 127.6.54.1:43137 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:54.590492 11559 connection.cc:513] server connection from 127.6.54.1:43795 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:55.619946 12022 batcher.cc:349] Timed out: Failed to write batch of 100 ops to tablet dd1b98d584d5486294a0219bf1a80f1b after 42 attempt(s): Failed to write to server: (no server available): Write(tablet: dd1b98d584d5486294a0219bf1a80f1b, num_ops: 100, num_attempts: 42) passed its deadline: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:55.620079 6360 ts_recovery-itest.cc:752] could not flush session: IO error: Some errors occurred I0408 18:33:55.620144 6360 ts_recovery-itest.cc:756] successfully detected TS crash! W0408 18:33:55.639734 12021 meta_cache.cc:219] Tablet dd1b98d584d5486294a0219bf1a80f1b: Replica 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1:43137) has failed: Network error: Client connection negotiation failed: client connection to 127.6.54.1:43137: connect: Connection refused (error 111) I0408 18:33:56.084270 12067 raft_consensus.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3502ea6d66a64ab2b1aaed18d282ebec) I0408 18:33:56.084311 12067 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Snoozing failure detection for 2.040s (starting election) I0408 18:33:56.084367 12067 raft_consensus.cc:458] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:56.084592 12067 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 3502ea6d66a64ab2b1aaed18d282ebec I0408 18:33:56.084648 12067 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6c9d88f121474dd99a68c2b254085723 W0408 18:33:56.084977 11878 leader_election.cc:287] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3502ea6d66a64ab2b1aaed18d282ebec: Network error: Client connection negotiation failed: client connection to 127.6.54.1:43137: connect: Connection refused (error 111) I0408 18:33:56.086208 12071 raft_consensus.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3502ea6d66a64ab2b1aaed18d282ebec) I0408 18:33:56.086249 12071 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Snoozing failure detection for 2.050s (starting election) I0408 18:33:56.086319 12071 raft_consensus.cc:458] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:56.086573 12071 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 3502ea6d66a64ab2b1aaed18d282ebec I0408 18:33:56.086743 12071 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 87aa5becfacc4cf39bdcb08f191990b8 W0408 18:33:56.087051 11754 leader_election.cc:287] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3502ea6d66a64ab2b1aaed18d282ebec: Network error: Client connection negotiation failed: client connection to 127.6.54.1:43137: connect: Connection refused (error 111) I0408 18:33:56.087594 11821 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dd1b98d584d5486294a0219bf1a80f1b" candidate_uuid: "87aa5becfacc4cf39bdcb08f191990b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5424 } } ignore_live_leader: false dest_uuid: "6c9d88f121474dd99a68c2b254085723" is_pre_election: true I0408 18:33:56.087630 11821 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Snoozing failure detection for 1.638s (vote granted) I0408 18:33:56.087641 11821 raft_consensus.cc:2215] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87aa5becfacc4cf39bdcb08f191990b8 in term 1. I0408 18:33:56.087834 11880 leader_election.cc:363] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 6c9d88f121474dd99a68c2b254085723 I0408 18:33:56.087860 11880 leader_election.cc:258] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0408 18:33:56.088366 12067 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Snoozing failure detection for 1.898s (election complete) I0408 18:33:56.088389 12067 raft_consensus.cc:2507] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0408 18:33:56.088397 12067 raft_consensus.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3502ea6d66a64ab2b1aaed18d282ebec) I0408 18:33:56.088402 12067 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Snoozing failure detection for 1.841s (starting election) I0408 18:33:56.088405 12067 raft_consensus.cc:2752] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:56.088997 12067 raft_consensus.cc:458] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:56.089148 12067 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 election: Requesting vote from peer 3502ea6d66a64ab2b1aaed18d282ebec I0408 18:33:56.089182 12067 leader_election.cc:231] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 election: Requesting vote from peer 6c9d88f121474dd99a68c2b254085723 W0408 18:33:56.089361 11878 leader_election.cc:287] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3502ea6d66a64ab2b1aaed18d282ebec: Network error: Client connection negotiation failed: client connection to 127.6.54.1:43137: connect: Connection refused (error 111) I0408 18:33:56.089629 11821 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dd1b98d584d5486294a0219bf1a80f1b" candidate_uuid: "87aa5becfacc4cf39bdcb08f191990b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5424 } } ignore_live_leader: false dest_uuid: "6c9d88f121474dd99a68c2b254085723" I0408 18:33:56.089661 11821 raft_consensus.cc:2752] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 1 FOLLOWER]: Advancing to term 2 I0408 18:33:56.089689 11821 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Snoozing failure detection for 1.690s (vote granted) I0408 18:33:56.089692 11945 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dd1b98d584d5486294a0219bf1a80f1b" candidate_uuid: "6c9d88f121474dd99a68c2b254085723" candidate_term: 2 candidate_status { last_received { term: 1 index: 5423 } } ignore_live_leader: false dest_uuid: "87aa5becfacc4cf39bdcb08f191990b8" is_pre_election: true I0408 18:33:56.089720 11945 raft_consensus.cc:2145] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c9d88f121474dd99a68c2b254085723 in current term 2: Already voted for candidate 87aa5becfacc4cf39bdcb08f191990b8 in this term. I0408 18:33:56.089967 11755 leader_election.cc:377] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 87aa5becfacc4cf39bdcb08f191990b8. Message: Invalid argument: T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c9d88f121474dd99a68c2b254085723 in current term 2: Already voted for candidate 87aa5becfacc4cf39bdcb08f191990b8 in this term. I0408 18:33:56.089990 11755 leader_election.cc:258] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0408 18:33:56.090157 11821 raft_consensus.cc:2215] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87aa5becfacc4cf39bdcb08f191990b8 in term 2. I0408 18:33:56.090214 12071 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723: Snoozing failure detection for 2.236s (election complete) I0408 18:33:56.090231 12071 raft_consensus.cc:2455] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0408 18:33:56.090343 11880 leader_election.cc:363] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 election: Vote granted by peer 6c9d88f121474dd99a68c2b254085723 I0408 18:33:56.090361 11880 leader_election.cc:258] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0408 18:33:56.090417 12067 raft_consensus.cc:2700] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8: Snoozing failure detection for 2.171s (election complete) I0408 18:33:56.090440 12067 raft_consensus.cc:2507] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 2 FOLLOWER]: Leader election won for term 2 I0408 18:33:56.090503 12067 raft_consensus.cc:572] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [term 2 LEADER]: Becoming Leader. State: Replica: 87aa5becfacc4cf39bdcb08f191990b8, State: Running, Role: LEADER I0408 18:33:56.090603 12067 consensus_queue.cc:203] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5423, Committed index: 5423, Last appended: 1.5424, Last appended by leader: 5424, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:56.091289 11568 catalog_manager.cc:3595] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 reported cstate change: term changed from 1 to 2, leader changed from 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1) to 87aa5becfacc4cf39bdcb08f191990b8 (127.6.54.3). New cstate: current_term: 2 leader_uuid: "87aa5becfacc4cf39bdcb08f191990b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } health_report { overall_health: UNKNOWN } } } W0408 18:33:56.094388 11878 consensus_peers.cc:435] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 -> Peer 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1:43137): Couldn't send request to peer 3502ea6d66a64ab2b1aaed18d282ebec for tablet dd1b98d584d5486294a0219bf1a80f1b. Status: Network error: Client connection negotiation failed: client connection to 127.6.54.1:43137: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0408 18:33:56.094425 11821 raft_consensus.cc:1058] T dd1b98d584d5486294a0219bf1a80f1b P 6c9d88f121474dd99a68c2b254085723 [term 2 FOLLOWER]: Refusing update from remote peer 87aa5becfacc4cf39bdcb08f191990b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5423. Preceding OpId from leader: term: 2 index: 5426. (index mismatch) I0408 18:33:56.094756 12075 consensus_queue.cc:962] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5425, Last known committed idx: 5423, Time since last communication: 0.000s W0408 18:33:56.096045 12084 rpcz_store.cc:251] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47868 (ReqId={client: b01ec804c702469dbecf2f0102cf1ffc, seq_no=5422, attempt_no=1}) took 1500ms (client timeout 1999). W0408 18:33:56.096678 12084 rpcz_store.cc:255] Trace: 0408 18:33:54.595968 (+ 0us) service_pool.cc:163] Inserting onto call queue 0408 18:33:54.596014 (+ 46us) service_pool.cc:222] Handling call 0408 18:33:56.096013 (+1499999us) inbound_call.cc:157] Queueing success response Metrics: {} I0408 18:33:56.219365 6360 external_mini_cluster.cc:809] Running /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.6.54.1:43137 --local_ip_for_outbound_sockets=127.6.54.1 --webserver_port=41447 --webserver_interface=127.6.54.1 --tserver_master_addrs=127.6.54.62:46549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --maintenance_manager_num_threads=3 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0408 18:33:56.254380 12089 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0408 18:33:56.254513 12089 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0408 18:33:56.255249 12089 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0408 18:33:56.255275 12089 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.54.1 I0408 18:33:56.256230 12089 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.6.54.1:43137 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb --webserver_interface=127.6.54.1 --webserver_port=41447 --tserver_master_addrs=127.6.54.62:46549 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.12089 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --maintenance_manager_num_threads=3 --enable_minidumps=false --local_ip_for_outbound_sockets=127.6.54.1 --log_dir=/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.17.0-SNAPSHOT revision 834a99bd863227e2c59278bd953b80acc7e18e4f build type RELEASE built by jenkins at 08 Apr 2019 11:04:39 PST on quasar-nlvoik-1.vpc.cloudera.com build id 305457 I0408 18:33:56.257243 12089 tablet_server_main.cc:79] Initializing tablet server... I0408 18:33:56.257472 12089 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 37011us I0408 18:33:56.257630 12089 fs_manager.cc:260] Metadata directory not provided I0408 18:33:56.257676 12089 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0408 18:33:56.258744 12089 fs_manager.cc:397] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I0408 18:33:56.258985 12089 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0408 18:33:56.259032 12089 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0408 18:33:56.261168 12089 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0408 18:33:56.261545 12089 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data,/tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/wal uuid: "3502ea6d66a64ab2b1aaed18d282ebec" format_stamp: "Formatted at 2019-04-08 18:33:50 on dist-test-slave-dist-test-slave-gxv9" I0408 18:33:56.261591 12089 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/data Total live blocks: 12 Total live bytes: 851414 Total live bytes (after alignment): 884736 Total number of LBM containers: 12 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0408 18:33:56.291777 12089 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0408 18:33:56.291975 12089 ts_tablet_manager.cc:287] Loading tablet metadata (0/1 complete) I0408 18:33:56.291996 12089 ts_tablet_manager.cc:915] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Loading tablet metadata I0408 18:33:56.292610 12089 ts_tablet_manager.cc:299] Loaded tablet metadata (1 live tablets) I0408 18:33:56.293118 12089 ts_tablet_manager.cc:1080] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Registered tablet (data state: TABLET_DATA_READY) I0408 18:33:56.293267 12106 ts_tablet_manager.cc:938] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Bootstrapping tablet I0408 18:33:56.293289 12089 tablet_server_main.cc:84] Starting tablet server... I0408 18:33:56.293360 12106 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Bootstrap starting. I0408 18:33:56.298344 12089 rpc_server.cc:201] RPC server started. Bound to: 127.6.54.1:43137 I0408 18:33:56.298429 12089 webserver.cc:173] Starting webserver on 127.6.54.1:41447 I0408 18:33:56.298446 12089 webserver.cc:184] Document root disabled I0408 18:33:56.298714 12089 webserver.cc:311] Webserver started. Bound to: http://127.6.54.1:41447/ I0408 18:33:56.298955 12089 server_base.cc:607] Dumped server information to /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb I0408 18:33:56.299296 12089 tablet_server_main.cc:87] Tablet server successfully started. I0408 18:33:56.299331 12212 process_memory.cc:182] Process hard memory limit is 11.744250 GB I0408 18:33:56.299347 12212 process_memory.cc:184] Process soft memory limit is 9.395400 GB I0408 18:33:56.299353 12212 process_memory.cc:187] Process memory pressure threshold is 7.046550 GB I0408 18:33:56.303021 12211 heartbeater.cc:346] Connected to a master server at 127.6.54.62:46549 I0408 18:33:56.303066 12211 heartbeater.cc:426] Registering TS with master... I0408 18:33:56.303143 12211 heartbeater.cc:475] Master 127.6.54.62:46549 requested a full tablet report, sending... I0408 18:33:56.303606 11568 ts_manager.cc:87] Re-registered known tserver with Master: 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1:43137) I0408 18:33:56.304232 11568 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.6.54.1:47943 I0408 18:33:56.305058 6360 external_mini_cluster.cc:871] Started /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver as pid 12089 I0408 18:33:56.345084 12106 log.cc:520] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Log is configured to *not* fsync() on all Append() calls W0408 18:33:56.483755 11878 consensus_peers.cc:435] T dd1b98d584d5486294a0219bf1a80f1b P 87aa5becfacc4cf39bdcb08f191990b8 -> Peer 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1:43137): Couldn't send request to peer 3502ea6d66a64ab2b1aaed18d282ebec for tablet dd1b98d584d5486294a0219bf1a80f1b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0408 18:33:56.771353 12106 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Bootstrap replayed 1/1 log segments. Stats: ops{read=5424 overwritten=0 applied=5423 ignored=4273} inserts{seen=14700 ignored=0} mutations{seen=1002 ignored=0} orphaned_commits=0. Pending: 1 replicates I0408 18:33:56.771692 12106 tablet_bootstrap.cc:436] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Bootstrap complete. I0408 18:33:56.772583 12106 ts_tablet_manager.cc:955] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Time spent bootstrapping tablet: real 0.479s user 0.421s sys 0.056s I0408 18:33:56.773629 12106 raft_consensus.cc:304] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:56.773979 12106 raft_consensus.cc:605] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3502ea6d66a64ab2b1aaed18d282ebec, State: Initialized, Role: FOLLOWER I0408 18:33:56.774103 12106 consensus_queue.cc:226] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5423, Last appended: 1.5424, Last appended by leader: 5424, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3502ea6d66a64ab2b1aaed18d282ebec" member_type: VOTER last_known_addr { host: "127.6.54.1" port: 43137 } } peers { permanent_uuid: "87aa5becfacc4cf39bdcb08f191990b8" member_type: VOTER last_known_addr { host: "127.6.54.3" port: 44449 } } peers { permanent_uuid: "6c9d88f121474dd99a68c2b254085723" member_type: VOTER last_known_addr { host: "127.6.54.2" port: 34299 } } I0408 18:33:56.774261 12106 ts_tablet_manager.cc:983] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s I0408 18:33:56.774261 12211 heartbeater.cc:467] Master 127.6.54.62:46549 was elected leader, sending a full tablet report... Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 74c5e6b7d8a34af28738dbbe1d654b2b | 127.6.54.62:46549 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 3502ea6d66a64ab2b1aaed18d282ebec | 127.6.54.1:43137 | HEALTHY 6c9d88f121474dd99a68c2b254085723 | 127.6.54.2:34299 | HEALTHY 87aa5becfacc4cf39bdcb08f191990b8 | 127.6.54.3:44449 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.6.54.1 | experimental | 127.6.54.1:43137 local_ip_for_outbound_sockets | 127.6.54.2 | experimental | 127.6.54.2:34299 local_ip_for_outbound_sockets | 127.6.54.3 | experimental | 127.6.54.3:44449 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb | hidden | 127.6.54.1:43137 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/info.pb | hidden | 127.6.54.2:34299 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/info.pb | hidden | 127.6.54.3:44449 Version Summary Version | Servers --------------------------+------------------------- 1.7.0-cdh5.17.0-SNAPSHOT | all 4 server(s) checked Tablet dd1b98d584d5486294a0219bf1a80f1b of table 'test-workload' is conflicted: Tablet dd1b98d584d5486294a0219bf1a80f1b of table 'test-workload' replicas' active configs disagree with the master's 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1:43137): RUNNING 87aa5becfacc4cf39bdcb08f191990b8 (127.6.54.3:44449): RUNNING [LEADER] 6c9d88f121474dd99a68c2b254085723 (127.6.54.2:34299): RUNNING All reported replicas are: A = 3502ea6d66a64ab2b1aaed18d282ebec B = 87aa5becfacc4cf39bdcb08f191990b8 C = 6c9d88f121474dd99a68c2b254085723 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | A B* C | 2 | -1 | Yes C | A B* C | 2 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0408 18:33:56.796983 6360 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0408 18:33:56.802908 12212 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 453782 bytes on disk I0408 18:33:56.802992 12212 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 453782 bytes on disk I0408 18:33:56.803107 12212 maintenance_manager.cc:302] P 3502ea6d66a64ab2b1aaed18d282ebec: Scheduling UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b): 453782 bytes on disk I0408 18:33:56.803279 12096 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4} I0408 18:33:56.803299 12095 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":5504} I0408 18:33:56.803282 12094 maintenance_manager.cc:492] P 3502ea6d66a64ab2b1aaed18d282ebec: UndoDeltaBlockGCOp(dd1b98d584d5486294a0219bf1a80f1b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 74c5e6b7d8a34af28738dbbe1d654b2b | 127.6.54.62:46549 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 3502ea6d66a64ab2b1aaed18d282ebec | 127.6.54.1:43137 | HEALTHY 6c9d88f121474dd99a68c2b254085723 | 127.6.54.2:34299 | HEALTHY 87aa5becfacc4cf39bdcb08f191990b8 | 127.6.54.3:44449 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.6.54.1 | experimental | 127.6.54.1:43137 local_ip_for_outbound_sockets | 127.6.54.2 | experimental | 127.6.54.2:34299 local_ip_for_outbound_sockets | 127.6.54.3 | experimental | 127.6.54.3:44449 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb | hidden | 127.6.54.1:43137 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/info.pb | hidden | 127.6.54.2:34299 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/info.pb | hidden | 127.6.54.3:44449 Version Summary Version | Servers --------------------------+------------------------- 1.7.0-cdh5.17.0-SNAPSHOT | all 4 server(s) checked Tablet dd1b98d584d5486294a0219bf1a80f1b of table 'test-workload' is conflicted: Tablet dd1b98d584d5486294a0219bf1a80f1b of table 'test-workload' replicas' active configs disagree with the master's 3502ea6d66a64ab2b1aaed18d282ebec (127.6.54.1:43137): RUNNING 87aa5becfacc4cf39bdcb08f191990b8 (127.6.54.3:44449): RUNNING [LEADER] 6c9d88f121474dd99a68c2b254085723 (127.6.54.2:34299): RUNNING All reported replicas are: A = 3502ea6d66a64ab2b1aaed18d282ebec B = 87aa5becfacc4cf39bdcb08f191990b8 C = 6c9d88f121474dd99a68c2b254085723 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | A B* C | 2 | -1 | Yes C | A B* C | 2 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0408 18:33:56.968454 6360 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0408 18:33:57.035642 12168 raft_consensus.cc:2752] T dd1b98d584d5486294a0219bf1a80f1b P 3502ea6d66a64ab2b1aaed18d282ebec [term 1 FOLLOWER]: Advancing to term 2 Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 74c5e6b7d8a34af28738dbbe1d654b2b | 127.6.54.62:46549 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 3502ea6d66a64ab2b1aaed18d282ebec | 127.6.54.1:43137 | HEALTHY 6c9d88f121474dd99a68c2b254085723 | 127.6.54.2:34299 | HEALTHY 87aa5becfacc4cf39bdcb08f191990b8 | 127.6.54.3:44449 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.6.54.1 | experimental | 127.6.54.1:43137 local_ip_for_outbound_sockets | 127.6.54.2 | experimental | 127.6.54.2:34299 local_ip_for_outbound_sockets | 127.6.54.3 | experimental | 127.6.54.3:44449 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-0/data/info.pb | hidden | 127.6.54.1:43137 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-1/data/info.pb | hidden | 127.6.54.2:34299 server_dump_info_path | /tmp/dist-test-task3Gp0Ia/test-tmp/ts_recovery-itest.0.DifferentFaultPoints_Kudu969Test.Test_1.1554748385377146-6360/minicluster-data/ts-2/data/info.pb | hidden | 127.6.54.3:44449 Version Summary Version | Servers --------------------------+------------------------- 1.7.0-cdh5.17.0-SNAPSHOT | all 4 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0408 18:33:57.215000 6360 log_verifier.cc:121] Checking tablet dd1b98d584d5486294a0219bf1a80f1b I0408 18:33:57.452543 6360 log_verifier.cc:172] Verified matching terms for 5428 ops in tablet dd1b98d584d5486294a0219bf1a80f1b I0408 18:33:57.454461 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 12089 W0408 18:33:57.457403 11878 connection.cc:513] client connection to 127.6.54.1:43137 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:57.457407 11547 connection.cc:513] client connection to 127.6.54.1:43137 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:57.457422 11559 connection.cc:513] server connection from 127.6.54.1:47943 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:57.457585 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 11743 W0408 18:33:57.460669 11879 connection.cc:513] server connection from 127.6.54.2:57531 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:57.460815 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-tserver with pid 11867 W0408 18:33:57.460817 11559 connection.cc:513] client connection to 127.6.54.2:34299 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:57.462420 11558 connection.cc:513] server connection from 127.6.54.2:38745 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:57.464210 11558 connection.cc:513] client connection to 127.6.54.3:44449 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0408 18:33:57.464268 6360 external_mini_cluster.cc:1001] Killing /tmp/dist-test-task3Gp0Ia/build/release/bin/kudu-master with pid 11548 W0408 18:33:57.466960 11993 connection.cc:513] client connection to 127.6.54.62:46549 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0408 18:33:57.466960 11547 connection.cc:513] client connection to 127.6.54.62:46549 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] DifferentFaultPoints/Kudu969Test.Test/1 (6736 ms) [----------] 2 tests from DifferentFaultPoints/Kudu969Test (12548 ms total) [----------] Global test environment tear-down [==========] 14 tests from 2 test cases ran. (52094 ms total) [ PASSED ] 13 tests. [ FAILED ] 1 test, listed below: [ FAILED ] BlockManagerType/TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks/1, where GetParam() = "log" 1 FAILED TEST