[dinesh@ve0518 debug]$ ./start-master.sh && ./start-tserver0.sh && ./start-tserver1.sh && ./start-tserver2.sh [dinesh@ve0518 debug]$ I0825 15:49:47.924329 7860 mem_tracker.cc:138] MemTracker: hard memory limit is 75.496170 GB I0825 15:49:47.924409 7860 mem_tracker.cc:140] MemTracker: soft memory limit is 45.297703 GB I0825 15:49:47.926229 7864 mem_tracker.cc:138] MemTracker: hard memory limit is 75.496170 GB I0825 15:49:47.926300 7864 mem_tracker.cc:140] MemTracker: soft memory limit is 45.297703 GB I0825 15:49:47.926425 7860 master_main.cc:58] Initializing master server... I0825 15:49:47.926435 7862 mem_tracker.cc:138] MemTracker: hard memory limit is 75.496170 GB I0825 15:49:47.926519 7862 mem_tracker.cc:140] MemTracker: soft memory limit is 45.297703 GB I0825 15:49:47.926641 7860 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 457876 I0825 15:49:47.927808 7866 mem_tracker.cc:138] MemTracker: hard memory limit is 75.496170 GB I0825 15:49:47.927887 7866 mem_tracker.cc:140] MemTracker: soft memory limit is 45.297703 GB I0825 15:49:47.928891 7864 tablet_server_main.cc:54] Initializing tablet server... I0825 15:49:47.929143 7862 tablet_server_main.cc:54] Initializing tablet server... I0825 15:49:47.929208 7864 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 457876 I0825 15:49:47.929522 7862 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 457876 I0825 15:49:47.930109 7866 tablet_server_main.cc:54] Initializing tablet server... I0825 15:49:47.930109 7860 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0 uuid: "3b313b752ce844209ce84009c91c94df" format_stamp: "Formatted at 2016-08-25 21:52:23 on ve0518.halxg.cloudera.com" I0825 15:49:47.930469 7866 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 457876 I0825 15:49:47.931610 7864 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1 uuid: "92f34d06b19146db82cf42701d2363c5" format_stamp: "Formatted at 2016-08-25 21:52:23 on ve0518.halxg.cloudera.com" I0825 15:49:47.932102 7862 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0 uuid: "d9caa1baaef84124a4a530b35945fad3" format_stamp: "Formatted at 2016-08-25 21:52:23 on ve0518.halxg.cloudera.com" I0825 15:49:47.932307 7860 master_main.cc:61] Starting Master server... I0825 15:49:47.932883 7866 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2 uuid: "401cb2671e94452f87838515c7408bdc" format_stamp: "Formatted at 2016-08-25 21:52:23 on ve0518.halxg.cloudera.com" I0825 15:49:47.933531 7864 ts_tablet_manager.cc:606] Loading metadata for tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:49:47.933840 7864 ts_tablet_manager.cc:606] Loading metadata for tablet 17aec689be0047b987494787029f6e08 I0825 15:49:47.933964 7864 ts_tablet_manager.cc:606] Loading metadata for tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:49:47.934123 7864 ts_tablet_manager.cc:764] Registered tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:49:47.934221 7862 ts_tablet_manager.cc:606] Loading metadata for tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:49:47.934278 7864 ts_tablet_manager.cc:764] Registered tablet 17aec689be0047b987494787029f6e08 I0825 15:49:47.934310 7864 ts_tablet_manager.cc:764] Registered tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:49:47.934355 7913 ts_tablet_manager.cc:630] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Bootstrapping tablet I0825 15:49:47.934432 7864 tablet_server_main.cc:57] Starting tablet server... I0825 15:49:47.934526 7913 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Bootstrap starting. I0825 15:49:47.934545 7862 ts_tablet_manager.cc:606] Loading metadata for tablet 17aec689be0047b987494787029f6e08 I0825 15:49:47.934721 7862 ts_tablet_manager.cc:606] Loading metadata for tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:49:47.934901 7862 ts_tablet_manager.cc:764] Registered tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:49:47.934907 7913 tablet_bootstrap.cc:563] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:47.934962 7866 ts_tablet_manager.cc:606] Loading metadata for tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:49:47.934993 7913 tablet_bootstrap.cc:619] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 I0825 15:49:47.935009 7913 tablet_bootstrap.cc:627] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery in preparation for log replay I0825 15:49:47.935062 7862 ts_tablet_manager.cc:764] Registered tablet 17aec689be0047b987494787029f6e08 I0825 15:49:47.935098 7862 ts_tablet_manager.cc:764] Registered tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:49:47.935140 7931 ts_tablet_manager.cc:630] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Bootstrapping tablet W0825 15:49:47.935184 7913 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:47.935195 7913 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:47.935225 7862 tablet_server_main.cc:57] Starting tablet server... I0825 15:49:47.935250 7866 ts_tablet_manager.cc:606] Loading metadata for tablet 17aec689be0047b987494787029f6e08 I0825 15:49:47.935343 7931 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Bootstrap starting. I0825 15:49:47.935405 7866 ts_tablet_manager.cc:606] Loading metadata for tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:49:47.935580 7866 ts_tablet_manager.cc:764] Registered tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:49:47.935744 7866 ts_tablet_manager.cc:764] Registered tablet 17aec689be0047b987494787029f6e08 I0825 15:49:47.935789 7866 ts_tablet_manager.cc:764] Registered tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:49:47.935843 7951 ts_tablet_manager.cc:630] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Bootstrapping tablet I0825 15:49:47.935853 7931 tablet_bootstrap.cc:563] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:47.935915 7866 tablet_server_main.cc:57] Starting tablet server... I0825 15:49:47.935961 7931 tablet_bootstrap.cc:619] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 I0825 15:49:47.935978 7931 tablet_bootstrap.cc:627] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery in preparation for log replay I0825 15:49:47.936027 7951 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Bootstrap starting. W0825 15:49:47.936159 7931 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:47.936177 7931 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:47.936381 7860 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:57007 I0825 15:49:47.936432 7951 tablet_bootstrap.cc:563] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:47.936472 7860 webserver.cc:122] Starting webserver on localhost:0 I0825 15:49:47.936480 7860 webserver.cc:133] Document root disabled I0825 15:49:47.936542 7951 tablet_bootstrap.cc:619] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 I0825 15:49:47.936563 7951 tablet_bootstrap.cc:627] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery in preparation for log replay I0825 15:49:47.936744 7913 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 for valid entry headers following offset 270... I0825 15:49:47.936759 7860 net_util.cc:195] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. W0825 15:49:47.936784 7951 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:47.936800 7951 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:47.936874 7860 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:50834/ I0825 15:49:47.937036 7860 server_base.cc:238] Dumped server information to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/info.pb I0825 15:49:47.937743 8006 tablet_bootstrap.cc:400] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Bootstrap starting. I0825 15:49:47.938019 7931 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 for valid entry headers following offset 296... I0825 15:49:47.938573 7951 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 for valid entry headers following offset 258... I0825 15:49:47.938618 8006 tablet_bootstrap.cc:563] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0825 15:49:47.938693 8006 tablet_bootstrap.cc:619] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 I0825 15:49:47.938722 8006 tablet_bootstrap.cc:627] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery in preparation for log replay W0825 15:49:47.938849 8006 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:47.938860 8006 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:47.940827 8006 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 for valid entry headers following offset 43110... I0825 15:49:47.942782 7864 rpc_server.cc:164] RPC server started. Bound to: 127.108.26.1:33958 I0825 15:49:47.942874 7864 webserver.cc:122] Starting webserver on 127.108.26.1:0 I0825 15:49:47.942880 7864 webserver.cc:133] Document root disabled I0825 15:49:47.943022 7864 webserver.cc:217] Webserver started. Bound to: http://127.108.26.1:41575/ I0825 15:49:47.943152 7864 server_base.cc:238] Dumped server information to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/info.pb I0825 15:49:47.943294 7864 tablet_server_main.cc:60] Tablet server successfully started. I0825 15:49:47.944677 8219 heartbeater.cc:294] Connected to a master server at 127.0.0.1:57007 I0825 15:49:47.944696 8219 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.944720 8219 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... I0825 15:49:47.944906 7862 rpc_server.cc:164] RPC server started. Bound to: 127.108.26.0:37291 I0825 15:49:47.945008 7862 webserver.cc:122] Starting webserver on 127.108.26.0:0 I0825 15:49:47.945014 7862 webserver.cc:133] Document root disabled W0825 15:49:47.945051 8219 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:47.945068 8219 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.945081 8219 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... I0825 15:49:47.945194 7862 webserver.cc:217] Webserver started. Bound to: http://127.108.26.0:38938/ W0825 15:49:47.945313 8219 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:47.945312 7866 rpc_server.cc:164] RPC server started. Bound to: 127.108.26.2:35954 I0825 15:49:47.945329 8219 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.945343 8219 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... I0825 15:49:47.945363 7862 server_base.cc:238] Dumped server information to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/info.pb I0825 15:49:47.945410 7866 webserver.cc:122] Starting webserver on 127.108.26.2:0 I0825 15:49:47.945417 7866 webserver.cc:133] Document root disabled I0825 15:49:47.945515 7862 tablet_server_main.cc:60] Tablet server successfully started. W0825 15:49:47.945572 8219 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:47.945577 7866 webserver.cc:217] Webserver started. Bound to: http://127.108.26.2:33225/ W0825 15:49:47.945582 8219 heartbeater.cc:325] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0825 15:49:47.945739 7866 server_base.cc:238] Dumped server information to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/info.pb I0825 15:49:47.945973 7866 tablet_server_main.cc:60] Tablet server successfully started. I0825 15:49:47.946547 8267 heartbeater.cc:294] Connected to a master server at 127.0.0.1:57007 I0825 15:49:47.946570 8267 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.946589 8267 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... I0825 15:49:47.946808 8270 heartbeater.cc:294] Connected to a master server at 127.0.0.1:57007 I0825 15:49:47.946825 8270 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.946841 8270 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... W0825 15:49:47.946920 8267 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:47.946943 8267 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.946959 8267 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... W0825 15:49:47.947077 8270 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:47.947093 8270 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.947105 8270 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... W0825 15:49:47.947227 8267 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:47.947243 8267 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.947257 8267 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... W0825 15:49:47.947322 8270 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:47.947337 8270 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.947348 8270 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... W0825 15:49:47.947474 8267 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 W0825 15:49:47.947484 8267 heartbeater.cc:325] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. W0825 15:49:47.947542 8270 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 W0825 15:49:47.947551 8270 heartbeater.cc:325] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0825 15:49:47.986500 7913 log_util.cc:606] Found no log entry headers I0825 15:49:47.986531 7913 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 258 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Prior entries: [off=168 REPLICATE (1.1)] [off=206 REPLICATE (50.2)] [off=232 COMMIT (1.1)] [off=258 COMMIT (50.2)] I0825 15:49:47.986549 7913 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 (valid entries through byte offset 258) I0825 15:49:47.986690 7913 tablet.cc:853] T f32f6e0b1f354643b6b030599a359fa6 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:47.987092 7913 log.cc:349] Log is configured to *not* fsync() on all Append() calls I0825 15:49:47.988082 7913 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:47.988101 7913 tablet_bootstrap.cc:1040] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: ReplayState: Previous OpId: 50.2, Committed OpId: 50.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:47.988339 7913 tablet_bootstrap.cc:656] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery I0825 15:49:47.988350 7913 tablet_bootstrap.cc:659] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387988349 I0825 15:49:47.988368 7913 tablet_bootstrap.cc:669] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387988349 I0825 15:49:47.988518 7913 tablet_bootstrap.cc:672] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387988349 I0825 15:49:47.988529 7913 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Bootstrap complete. I0825 15:49:47.989346 8006 log_util.cc:606] Found no log entry headers I0825 15:49:47.989365 8006 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 43098 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001: Prior entries: [off=42480 REPLICATE (0.74)] [off=42516 COMMIT (0.74)] [off=43062 REPLICATE (0.75)] [off=43098 COMMIT (0.75)] I0825 15:49:47.989377 8006 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 (valid entries through byte offset 43098) I0825 15:49:47.989487 8006 tablet.cc:853] T 00000000000000000000000000000000 Rewinding schema during bootstrap to Schema [ 10:entry_type[int8 NOT NULL], 11:entry_id[string NOT NULL], 12:metadata[string NOT NULL] ] I0825 15:49:47.989821 8006 log.cc:349] Log is configured to *not* fsync() on all Append() calls I0825 15:49:47.995164 7913 ts_tablet_manager.cc:635] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Time spent bootstrapping tablet: real 0.061s user 0.022s sys 0.037s I0825 15:49:47.995616 7913 raft_consensus.cc:275] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:49:47.995662 7913 raft_consensus.cc:503] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 50 index: 2 Committed: term: 50 index: 2} I0825 15:49:47.995684 7913 consensus_queue.cc:162] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 50.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:47.995921 7913 ts_tablet_manager.cc:662] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0825 15:49:47.995954 7913 ts_tablet_manager.cc:630] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Bootstrapping tablet I0825 15:49:47.996035 7951 log_util.cc:606] Found no log entry headers I0825 15:49:47.996058 7951 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 246 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Prior entries: [off=194 REPLICATE (1.1)] [off=194 REPLICATE (50.2)] [off=220 COMMIT (1.1)] [off=246 COMMIT (50.2)] I0825 15:49:47.996059 7913 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Bootstrap starting. I0825 15:49:47.996073 7951 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 (valid entries through byte offset 246) I0825 15:49:47.996181 7951 tablet.cc:853] T f32f6e0b1f354643b6b030599a359fa6 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:47.996310 7913 tablet_bootstrap.cc:563] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:47.996336 8219 heartbeater.cc:294] Connected to a master server at 127.0.0.1:57007 I0825 15:49:47.996354 8219 heartbeater.cc:361] Registering TS with master... I0825 15:49:47.996358 7913 tablet_bootstrap.cc:619] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 I0825 15:49:47.996366 7913 tablet_bootstrap.cc:627] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery in preparation for log replay I0825 15:49:47.996373 8219 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... W0825 15:49:47.996469 7913 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:47.996480 7913 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:47.996521 7913 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 for valid entry headers following offset 258... I0825 15:49:47.996548 7931 log_util.cc:606] Found no log entry headers I0825 15:49:47.996567 7931 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 284 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Prior entries: [off=232 REPLICATE (1.1)] [off=232 REPLICATE (50.2)] [off=258 COMMIT (1.1)] [off=284 COMMIT (50.2)] I0825 15:49:47.996577 8006 tablet_bootstrap.cc:400] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=74} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:47.996716 7931 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 (valid entries through byte offset 284) W0825 15:49:47.996721 8219 heartbeater.cc:455] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:47.996580 7951 log.cc:349] Log is configured to *not* fsync() on all Append() calls I0825 15:49:47.996736 8006 tablet_bootstrap.cc:1040] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: ReplayState: Previous OpId: 0.75, Committed OpId: 0.75, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:47.996799 7931 tablet.cc:853] T f32f6e0b1f354643b6b030599a359fa6 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:47.996984 8006 tablet_bootstrap.cc:656] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery I0825 15:49:47.996995 8006 tablet_bootstrap.cc:659] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery-1472165387996994 I0825 15:49:47.997030 8006 tablet_bootstrap.cc:669] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery-1472165387996994 I0825 15:49:47.997156 8006 tablet_bootstrap.cc:672] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery-1472165387996994 I0825 15:49:47.997160 7931 log.cc:349] Log is configured to *not* fsync() on all Append() calls I0825 15:49:47.997167 8006 tablet_bootstrap.cc:400] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Bootstrap complete. I0825 15:49:47.997699 7951 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:47.997720 7951 tablet_bootstrap.cc:1040] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: ReplayState: Previous OpId: 50.2, Committed OpId: 50.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:47.998000 7931 tablet_bootstrap.cc:833] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Overwriting operations starting at: 49.1 up to: 49.1 with operation: 1.1 I0825 15:49:47.998029 7951 tablet_bootstrap.cc:656] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery I0825 15:49:47.998042 7951 tablet_bootstrap.cc:659] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387998041 I0825 15:49:47.998077 7951 tablet_bootstrap.cc:669] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387998041 I0825 15:49:47.998102 7931 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=1 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:47.998121 7931 tablet_bootstrap.cc:1040] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: ReplayState: Previous OpId: 50.2, Committed OpId: 50.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:47.998208 7951 tablet_bootstrap.cc:672] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387998041 I0825 15:49:47.998221 7951 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Bootstrap complete. I0825 15:49:47.998342 7931 tablet_bootstrap.cc:656] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery I0825 15:49:47.998365 7931 tablet_bootstrap.cc:659] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387998353 I0825 15:49:47.998386 7931 tablet_bootstrap.cc:669] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387998353 I0825 15:49:47.998492 7931 tablet_bootstrap.cc:672] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472165387998353 I0825 15:49:47.998503 7931 tablet_bootstrap.cc:400] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Bootstrap complete. I0825 15:49:48.004494 8006 raft_consensus.cc:275] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b313b752ce844209ce84009c91c94df" member_type: VOTER } I0825 15:49:48.004539 8006 raft_consensus.cc:503] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b313b752ce844209ce84009c91c94df, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 75 Committed: term: 0 index: 75} I0825 15:49:48.004559 8006 consensus_queue.cc:162] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.75, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.004570 8006 raft_consensus.cc:320] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0825 15:49:48.004575 8006 raft_consensus.cc:376] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:49:48.004581 8006 raft_consensus.cc:2022] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 1 FOLLOWER]: Advancing to term 2 I0825 15:49:48.004766 8006 raft_consensus.cc:1973] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.682s I0825 15:49:48.004791 8006 raft_consensus.cc:392] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b313b752ce844209ce84009c91c94df" member_type: VOTER } I0825 15:49:48.004947 8006 leader_election.cc:248] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0825 15:49:48.005177 8286 raft_consensus.cc:1973] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.805s I0825 15:49:48.005213 8286 raft_consensus.cc:1863] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 2 FOLLOWER]: Leader election won for term 2 I0825 15:49:48.005460 8286 raft_consensus.cc:468] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 2 LEADER]: Becoming Leader. State: Replica: 3b313b752ce844209ce84009c91c94df, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 75 Committed: term: 0 index: 75} I0825 15:49:48.005516 8286 consensus_queue.cc:145] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.75, Committed index: 0.75, Last appended: 0.75, Current term: 2, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b313b752ce844209ce84009c91c94df" member_type: VOTER } I0825 15:49:48.005674 8006 sys_catalog.cc:255] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 2 leader_uuid: "3b313b752ce844209ce84009c91c94df" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b313b752ce844209ce84009c91c94df" member_type: VOTER } } I0825 15:49:48.005687 8006 sys_catalog.cc:258] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: This master's current role is: LEADER I0825 15:49:48.005697 8287 sys_catalog.cc:255] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b313b752ce844209ce84009c91c94df" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b313b752ce844209ce84009c91c94df" member_type: VOTER } } I0825 15:49:48.005705 8288 sys_catalog.cc:255] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b313b752ce844209ce84009c91c94df. Latest consensus state: current_term: 2 leader_uuid: "3b313b752ce844209ce84009c91c94df" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b313b752ce844209ce84009c91c94df" member_type: VOTER } } I0825 15:49:48.005733 8287 sys_catalog.cc:258] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: This master's current role is: LEADER I0825 15:49:48.005755 8288 sys_catalog.cc:258] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: This master's current role is: LEADER I0825 15:49:48.006042 8290 raft_consensus_state.cc:536] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [term 2 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 75 New committed index is: term: 2 index: 76 I0825 15:49:48.006067 8006 sys_catalog.cc:333] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: configured and running, proceeding with master startup. I0825 15:49:48.006078 8289 catalog_manager.cc:660] Loading table and tablet metadata into memory... W0825 15:49:48.006153 8291 catalog_manager.cc:365] Catalog manager background task thread going to sleep: Service unavailable: Catalog manager is not initialized. State: 1 I0825 15:49:48.006186 7860 master_main.cc:64] Master server successfully started. I0825 15:49:48.007199 8289 catalog_manager.cc:227] Loaded metadata for table test-table [id=162312d1479340109aec6ff77ce314f1] I0825 15:49:48.007673 8289 catalog_manager.cc:277] Loaded metadata for tablet 17aec689be0047b987494787029f6e08 (table test-table [id=162312d1479340109aec6ff77ce314f1]) I0825 15:49:48.007782 8289 catalog_manager.cc:277] Loaded metadata for tablet 1f7a6bd64c604f7fada909bd575708cd (table test-table [id=162312d1479340109aec6ff77ce314f1]) I0825 15:49:48.007870 8289 catalog_manager.cc:277] Loaded metadata for tablet f32f6e0b1f354643b6b030599a359fa6 (table test-table [id=162312d1479340109aec6ff77ce314f1]) I0825 15:49:48.008683 7951 ts_tablet_manager.cc:635] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Time spent bootstrapping tablet: real 0.073s user 0.023s sys 0.048s I0825 15:49:48.009021 7931 ts_tablet_manager.cc:635] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Time spent bootstrapping tablet: real 0.074s user 0.023s sys 0.050s I0825 15:49:48.009287 7951 raft_consensus.cc:275] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:49:48.009333 7951 raft_consensus.cc:503] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401cb2671e94452f87838515c7408bdc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 50 index: 2 Committed: term: 50 index: 2} I0825 15:49:48.009357 7951 consensus_queue.cc:162] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 50.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.009557 7931 raft_consensus.cc:275] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:49:48.009598 7931 raft_consensus.cc:503] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: FOLLOWER Watermarks: {Received: term: 50 index: 2 Committed: term: 50 index: 2} I0825 15:49:48.009619 7931 consensus_queue.cc:162] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 50.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.009624 7951 ts_tablet_manager.cc:662] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s I0825 15:49:48.009693 7951 ts_tablet_manager.cc:630] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Bootstrapping tablet I0825 15:49:48.009811 7951 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Bootstrap starting. I0825 15:49:48.009860 7931 ts_tablet_manager.cc:662] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0825 15:49:48.009932 7931 ts_tablet_manager.cc:630] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Bootstrapping tablet I0825 15:49:48.010032 7931 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Bootstrap starting. I0825 15:49:48.010076 7951 tablet_bootstrap.cc:563] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:48.010074 8270 heartbeater.cc:294] Connected to a master server at 127.0.0.1:57007 I0825 15:49:48.010105 8270 heartbeater.cc:361] Registering TS with master... I0825 15:49:48.010125 8270 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... I0825 15:49:48.010133 7951 tablet_bootstrap.cc:619] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 I0825 15:49:48.010144 7951 tablet_bootstrap.cc:627] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery in preparation for log replay I0825 15:49:48.010186 8267 heartbeater.cc:294] Connected to a master server at 127.0.0.1:57007 I0825 15:49:48.010210 8267 heartbeater.cc:361] Registering TS with master... I0825 15:49:48.010236 8267 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... W0825 15:49:48.010246 7951 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:48.010257 7951 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:48.010279 7931 tablet_bootstrap.cc:563] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:48.010295 7951 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 for valid entry headers following offset 258... I0825 15:49:48.010330 7931 tablet_bootstrap.cc:619] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 I0825 15:49:48.010331 7905 ts_manager.cc:83] Registered new tserver permanent_uuid: "401cb2671e94452f87838515c7408bdc" instance_seqno: 1472165387943440 with Master I0825 15:49:48.010339 7931 tablet_bootstrap.cc:627] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery in preparation for log replay W0825 15:49:48.010432 7931 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:48.010431 7902 ts_manager.cc:83] Registered new tserver permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" instance_seqno: 1472165387942958 with Master I0825 15:49:48.010442 7931 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:48.010824 7931 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 for valid entry headers following offset 270... I0825 15:49:48.014313 8267 heartbeater.cc:367] Master 127.0.0.1:57007 was elected leader, sending a full tablet report... I0825 15:49:48.043552 7913 log_util.cc:606] Found no log entry headers I0825 15:49:48.043581 7913 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 246 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Prior entries: [off=194 REPLICATE (1.1)] [off=194 REPLICATE (49.2)] [off=220 COMMIT (1.1)] [off=246 COMMIT (49.2)] I0825 15:49:48.043596 7913 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 (valid entries through byte offset 246) I0825 15:49:48.043676 7913 tablet.cc:853] T 17aec689be0047b987494787029f6e08 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:48.044714 7913 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:48.044737 7913 tablet_bootstrap.cc:1040] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: ReplayState: Previous OpId: 49.2, Committed OpId: 49.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:48.044942 7913 tablet_bootstrap.cc:656] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery I0825 15:49:48.044953 7913 tablet_bootstrap.cc:659] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery-1472165388044952 I0825 15:49:48.044973 7913 tablet_bootstrap.cc:669] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery-1472165388044952 I0825 15:49:48.045099 7913 tablet_bootstrap.cc:672] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery-1472165388044952 I0825 15:49:48.045110 7913 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Bootstrap complete. I0825 15:49:48.051923 7913 ts_tablet_manager.cc:635] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Time spent bootstrapping tablet: real 0.056s user 0.022s sys 0.034s I0825 15:49:48.052342 7913 raft_consensus.cc:275] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:49:48.052372 7913 raft_consensus.cc:503] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 49 index: 2 Committed: term: 49 index: 2} I0825 15:49:48.052392 7913 consensus_queue.cc:162] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 49.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.052613 7913 ts_tablet_manager.cc:662] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0825 15:49:48.052672 7913 ts_tablet_manager.cc:630] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Bootstrapping tablet I0825 15:49:48.052772 7913 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Bootstrap starting. I0825 15:49:48.053033 7913 tablet_bootstrap.cc:563] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:48.053104 7913 tablet_bootstrap.cc:619] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 I0825 15:49:48.053113 7913 tablet_bootstrap.cc:627] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery in preparation for log replay I0825 15:49:48.053117 8219 heartbeater.cc:294] Connected to a master server at 127.0.0.1:57007 I0825 15:49:48.053134 8219 heartbeater.cc:361] Registering TS with master... I0825 15:49:48.053151 8219 heartbeater.cc:375] Master 127.0.0.1:57007 requested a full tablet report, sending... W0825 15:49:48.053196 7913 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:48.053206 7913 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:48.053253 7913 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 for valid entry headers following offset 270... I0825 15:49:48.053436 7902 ts_manager.cc:83] Registered new tserver permanent_uuid: "92f34d06b19146db82cf42701d2363c5" instance_seqno: 1472165387941030 with Master I0825 15:49:48.061758 7951 log_util.cc:606] Found no log entry headers I0825 15:49:48.061784 7951 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 246 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Prior entries: [off=194 REPLICATE (1.1)] [off=194 REPLICATE (49.2)] [off=220 COMMIT (1.1)] [off=246 COMMIT (49.2)] I0825 15:49:48.061799 7951 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 (valid entries through byte offset 246) I0825 15:49:48.061874 7951 tablet.cc:853] T 17aec689be0047b987494787029f6e08 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:48.062700 7931 log_util.cc:606] Found no log entry headers I0825 15:49:48.062713 7931 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 258 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Prior entries: [off=168 REPLICATE (1.1)] [off=206 REPLICATE (49.2)] [off=232 COMMIT (1.1)] [off=258 COMMIT (49.2)] I0825 15:49:48.062723 7931 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 (valid entries through byte offset 258) I0825 15:49:48.062777 7931 tablet.cc:853] T 17aec689be0047b987494787029f6e08 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:48.062959 7951 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:48.062981 7951 tablet_bootstrap.cc:1040] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: ReplayState: Previous OpId: 49.2, Committed OpId: 49.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:48.063194 7951 tablet_bootstrap.cc:656] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery I0825 15:49:48.063205 7951 tablet_bootstrap.cc:659] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery-1472165388063204 I0825 15:49:48.063238 7951 tablet_bootstrap.cc:669] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery-1472165388063204 I0825 15:49:48.063340 7951 tablet_bootstrap.cc:672] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery-1472165388063204 I0825 15:49:48.063352 7951 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Bootstrap complete. I0825 15:49:48.063742 7931 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:48.063765 7931 tablet_bootstrap.cc:1040] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: ReplayState: Previous OpId: 49.2, Committed OpId: 49.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:48.063938 7931 tablet_bootstrap.cc:656] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery I0825 15:49:48.063949 7931 tablet_bootstrap.cc:659] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery-1472165388063948 I0825 15:49:48.063967 7931 tablet_bootstrap.cc:669] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery-1472165388063948 I0825 15:49:48.064059 7931 tablet_bootstrap.cc:672] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery-1472165388063948 I0825 15:49:48.064067 7931 tablet_bootstrap.cc:400] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Bootstrap complete. I0825 15:49:48.072557 7951 ts_tablet_manager.cc:635] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.043s I0825 15:49:48.072980 7951 raft_consensus.cc:275] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:49:48.073012 7951 raft_consensus.cc:503] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401cb2671e94452f87838515c7408bdc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 49 index: 2 Committed: term: 49 index: 2} I0825 15:49:48.073032 7951 consensus_queue.cc:162] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 49.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.073230 8270 heartbeater.cc:367] Master 127.0.0.1:57007 was elected leader, sending a full tablet report... I0825 15:49:48.073249 7951 ts_tablet_manager.cc:662] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0825 15:49:48.073292 7951 ts_tablet_manager.cc:630] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Bootstrapping tablet I0825 15:49:48.073384 7951 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Bootstrap starting. I0825 15:49:48.073401 7931 ts_tablet_manager.cc:635] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Time spent bootstrapping tablet: real 0.063s user 0.020s sys 0.042s I0825 15:49:48.073639 7951 tablet_bootstrap.cc:563] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:48.073690 7951 tablet_bootstrap.cc:619] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 I0825 15:49:48.073699 7951 tablet_bootstrap.cc:627] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery in preparation for log replay I0825 15:49:48.073799 7931 raft_consensus.cc:275] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } W0825 15:49:48.073814 7951 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:48.073823 7951 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:48.073828 7931 raft_consensus.cc:503] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: FOLLOWER Watermarks: {Received: term: 49 index: 2 Committed: term: 49 index: 2} I0825 15:49:48.073859 7931 consensus_queue.cc:162] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 49.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.073873 7951 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 for valid entry headers following offset 258... I0825 15:49:48.074023 7931 ts_tablet_manager.cc:662] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0825 15:49:48.074043 7931 ts_tablet_manager.cc:630] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Bootstrapping tablet I0825 15:49:48.074128 7931 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Bootstrap starting. I0825 15:49:48.074368 7931 tablet_bootstrap.cc:563] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:49:48.074415 7931 tablet_bootstrap.cc:619] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 I0825 15:49:48.074424 7931 tablet_bootstrap.cc:627] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery in preparation for log replay W0825 15:49:48.074520 7931 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:49:48.074530 7931 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:49:48.074568 7931 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 for valid entry headers following offset 296... I0825 15:49:48.097137 7913 log_util.cc:606] Found no log entry headers I0825 15:49:48.097163 7913 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 258 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Prior entries: [off=168 REPLICATE (1.1)] [off=206 REPLICATE (57.2)] [off=232 COMMIT (1.1)] [off=258 COMMIT (57.2)] I0825 15:49:48.097177 7913 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 (valid entries through byte offset 258) I0825 15:49:48.097254 7913 tablet.cc:853] T 1f7a6bd64c604f7fada909bd575708cd Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:48.098285 7913 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:48.098302 7913 tablet_bootstrap.cc:1040] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: ReplayState: Previous OpId: 57.2, Committed OpId: 57.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:48.098508 7913 tablet_bootstrap.cc:656] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery I0825 15:49:48.098520 7913 tablet_bootstrap.cc:659] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388098518 I0825 15:49:48.098537 7913 tablet_bootstrap.cc:669] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388098518 I0825 15:49:48.098711 7913 tablet_bootstrap.cc:672] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388098518 I0825 15:49:48.098722 7913 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Bootstrap complete. I0825 15:49:48.105147 7913 ts_tablet_manager.cc:635] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.033s I0825 15:49:48.105537 7913 raft_consensus.cc:275] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:49:48.105566 7913 raft_consensus.cc:503] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 57 index: 2 Committed: term: 57 index: 2} I0825 15:49:48.105586 7913 consensus_queue.cc:162] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 57.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.105764 8219 heartbeater.cc:367] Master 127.0.0.1:57007 was elected leader, sending a full tablet report... I0825 15:49:48.105819 7913 ts_tablet_manager.cc:662] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0825 15:49:48.121613 7951 log_util.cc:606] Found no log entry headers I0825 15:49:48.121634 7951 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 246 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Prior entries: [off=194 REPLICATE (1.1)] [off=194 REPLICATE (57.2)] [off=220 COMMIT (1.1)] [off=246 COMMIT (57.2)] I0825 15:49:48.121646 7951 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 (valid entries through byte offset 246) I0825 15:49:48.121721 7951 tablet.cc:853] T 1f7a6bd64c604f7fada909bd575708cd Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:48.122763 7951 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:48.122787 7951 tablet_bootstrap.cc:1040] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: ReplayState: Previous OpId: 57.2, Committed OpId: 57.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:48.122979 7951 tablet_bootstrap.cc:656] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery I0825 15:49:48.122989 7951 tablet_bootstrap.cc:659] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388122988 I0825 15:49:48.123008 7951 tablet_bootstrap.cc:669] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388122988 I0825 15:49:48.123080 7931 log_util.cc:606] Found no log entry headers I0825 15:49:48.123092 7931 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 284 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Prior entries: [off=232 REPLICATE (1.1)] [off=232 REPLICATE (57.2)] [off=258 COMMIT (1.1)] [off=284 COMMIT (57.2)] I0825 15:49:48.123097 7951 tablet_bootstrap.cc:672] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388122988 I0825 15:49:48.123101 7931 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 (valid entries through byte offset 284) I0825 15:49:48.123108 7951 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Bootstrap complete. I0825 15:49:48.123149 7931 tablet.cc:853] T 1f7a6bd64c604f7fada909bd575708cd Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:49:48.123999 7931 tablet_bootstrap.cc:833] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Overwriting operations starting at: 56.1 up to: 56.1 with operation: 1.1 I0825 15:49:48.124073 7931 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=1 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:49:48.124099 7931 tablet_bootstrap.cc:1040] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: ReplayState: Previous OpId: 57.2, Committed OpId: 57.2, Pending Replicates: 0, Pending Commits: 0 I0825 15:49:48.124272 7931 tablet_bootstrap.cc:656] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery I0825 15:49:48.124282 7931 tablet_bootstrap.cc:659] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388124282 I0825 15:49:48.124300 7931 tablet_bootstrap.cc:669] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388124282 I0825 15:49:48.124382 7931 tablet_bootstrap.cc:672] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472165388124282 I0825 15:49:48.124392 7931 tablet_bootstrap.cc:400] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Bootstrap complete. I0825 15:49:48.132529 7951 ts_tablet_manager.cc:635] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Time spent bootstrapping tablet: real 0.059s user 0.020s sys 0.038s I0825 15:49:48.132972 7951 raft_consensus.cc:275] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 57 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:49:48.133002 7951 raft_consensus.cc:503] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 57 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401cb2671e94452f87838515c7408bdc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 57 index: 2 Committed: term: 57 index: 2} I0825 15:49:48.133019 7951 consensus_queue.cc:162] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 57.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.133252 7951 ts_tablet_manager.cc:662] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s I0825 15:49:48.133949 7931 ts_tablet_manager.cc:635] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.044s I0825 15:49:48.134367 7931 raft_consensus.cc:275] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 57 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:49:48.134394 7931 raft_consensus.cc:503] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 57 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: FOLLOWER Watermarks: {Received: term: 57 index: 2 Committed: term: 57 index: 2} I0825 15:49:48.134420 7931 consensus_queue.cc:162] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 57.2, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:49:48.134629 7931 ts_tablet_manager.cc:662] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0825 15:49:48.182786 7868 delta_compaction.cc:259] Starting major delta compaction for columns metadata[string NOT NULL] I0825 15:49:48.182817 7868 delta_compaction.cc:263] Preparing to major compact delta file: 482005392666535073 I0825 15:49:48.182947 7868 multi_column_writer.cc:85] Opened CFile writer for column metadata[string NOT NULL] I0825 15:49:48.189944 7868 delta_compaction.cc:269] Finished major delta compaction of columns metadata[string NOT NULL] I0825 15:49:48.192095 7868 maintenance_manager.cc:353] Time spent running MajorDeltaCompactionOp(00000000000000000000000000000000): real 0.009s user 0.001s sys 0.000s I0825 15:49:49.580644 8309 raft_consensus.cc:376] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:49:49.580667 8309 raft_consensus.cc:2022] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Advancing to term 50 I0825 15:49:49.581017 8309 raft_consensus.cc:1973] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.581s I0825 15:49:49.581049 8309 raft_consensus.cc:392] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:49:49.581310 8309 leader_election.cc:223] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 50 election: Requesting vote from peer 92f34d06b19146db82cf42701d2363c5 I0825 15:49:49.581344 8309 leader_election.cc:223] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 50 election: Requesting vote from peer d9caa1baaef84124a4a530b35945fad3 I0825 15:49:49.582447 8090 raft_consensus.cc:2022] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Advancing to term 50 I0825 15:49:49.582451 8151 raft_consensus.cc:2022] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Advancing to term 50 I0825 15:49:49.582797 8151 raft_consensus.cc:1973] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.506s I0825 15:49:49.582809 8090 raft_consensus.cc:1973] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.544s I0825 15:49:49.583122 8151 raft_consensus.cc:1679] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401cb2671e94452f87838515c7408bdc in term 50. I0825 15:49:49.583149 8090 raft_consensus.cc:1679] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401cb2671e94452f87838515c7408bdc in term 50. I0825 15:49:49.583246 7914 leader_election.cc:347] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 50 election: Vote granted by peer d9caa1baaef84124a4a530b35945fad3 I0825 15:49:49.583261 7914 leader_election.cc:248] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. I0825 15:49:49.583283 7920 leader_election.cc:347] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 50 election: Vote granted by peer 92f34d06b19146db82cf42701d2363c5 I0825 15:49:49.583420 8330 raft_consensus.cc:1973] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.557s I0825 15:49:49.583441 8330 raft_consensus.cc:1863] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Leader election won for term 50 I0825 15:49:49.583694 8330 raft_consensus.cc:468] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 50 LEADER]: Becoming Leader. State: Replica: 401cb2671e94452f87838515c7408bdc, State: 1, Role: LEADER Watermarks: {Received: term: 49 index: 2 Committed: term: 49 index: 2} I0825 15:49:49.583786 8330 consensus_queue.cc:145] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 49.2, Committed index: 49.2, Last appended: 49.2, Current term: 50, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } W0825 15:49:49.584823 8334 raft_consensus_state.cc:543] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 50 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 49 index: 2, New majority replicated is: term: 49 index: 2, Current term is: 50 I0825 15:49:49.584913 7902 catalog_manager.cc:2018] T 17aec689be0047b987494787029f6e08 reported consensus state change: term changed from 49 to 50, leader changed from d9caa1baaef84124a4a530b35945fad3 (127.108.26.0) to 401cb2671e94452f87838515c7408bdc (127.108.26.2). New consensus state: current_term: 50 leader_uuid: "401cb2671e94452f87838515c7408bdc" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } } I0825 15:49:49.615257 8279 raft_consensus.cc:376] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:49:49.615274 8279 raft_consensus.cc:2022] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Advancing to term 51 I0825 15:49:49.615509 8279 raft_consensus.cc:1973] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 51 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.646s I0825 15:49:49.615541 8279 raft_consensus.cc:392] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 51 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:49:49.615945 8279 leader_election.cc:223] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 51 election: Requesting vote from peer d9caa1baaef84124a4a530b35945fad3 I0825 15:49:49.615990 8279 leader_election.cc:223] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 51 election: Requesting vote from peer 401cb2671e94452f87838515c7408bdc I0825 15:49:49.616873 8151 raft_consensus.cc:2022] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Advancing to term 51 I0825 15:49:49.616950 8170 raft_consensus.cc:2022] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Advancing to term 51 I0825 15:49:49.617198 8151 raft_consensus.cc:1973] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 51 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.550s I0825 15:49:49.617281 8170 raft_consensus.cc:1973] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 51 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.617s I0825 15:49:49.617491 8170 raft_consensus.cc:1679] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f34d06b19146db82cf42701d2363c5 in term 51. I0825 15:49:49.617507 8151 raft_consensus.cc:1679] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f34d06b19146db82cf42701d2363c5 in term 51. I0825 15:49:49.617655 7889 leader_election.cc:347] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 51 election: Vote granted by peer 401cb2671e94452f87838515c7408bdc I0825 15:49:49.617689 7889 leader_election.cc:248] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. I0825 15:49:49.617705 7886 leader_election.cc:347] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 51 election: Vote granted by peer d9caa1baaef84124a4a530b35945fad3 I0825 15:49:49.617913 8340 raft_consensus.cc:1973] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 51 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.578s I0825 15:49:49.617943 8340 raft_consensus.cc:1863] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 51 FOLLOWER]: Leader election won for term 51 I0825 15:49:49.618177 8340 raft_consensus.cc:468] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 51 LEADER]: Becoming Leader. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: LEADER Watermarks: {Received: term: 50 index: 2 Committed: term: 50 index: 2} I0825 15:49:49.618252 8340 consensus_queue.cc:145] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 50.2, Committed index: 50.2, Last appended: 50.2, Current term: 51, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } W0825 15:49:49.619187 8344 raft_consensus_state.cc:543] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 51 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 50 index: 2, New majority replicated is: term: 50 index: 2, Current term is: 51 I0825 15:49:49.619201 7902 catalog_manager.cc:2018] T f32f6e0b1f354643b6b030599a359fa6 reported consensus state change: term changed from 50 to 51. New consensus state: current_term: 51 leader_uuid: "92f34d06b19146db82cf42701d2363c5" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } } I0825 15:49:49.707619 8324 raft_consensus.cc:376] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 57 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:49:49.707633 8324 raft_consensus.cc:2022] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 57 FOLLOWER]: Advancing to term 58 I0825 15:49:49.707824 8324 raft_consensus.cc:1973] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 58 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.520s I0825 15:49:49.707851 8324 raft_consensus.cc:392] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 58 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:49:49.708204 8324 leader_election.cc:223] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 58 election: Requesting vote from peer 401cb2671e94452f87838515c7408bdc I0825 15:49:49.708235 8324 leader_election.cc:223] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 58 election: Requesting vote from peer 92f34d06b19146db82cf42701d2363c5 I0825 15:49:49.709020 8170 raft_consensus.cc:2022] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 57 FOLLOWER]: Advancing to term 58 I0825 15:49:49.709334 8090 raft_consensus.cc:2022] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 FOLLOWER]: Advancing to term 58 I0825 15:49:49.709465 8170 raft_consensus.cc:1973] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 58 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.519s I0825 15:49:49.709641 8090 raft_consensus.cc:1973] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 58 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.595s I0825 15:49:49.709744 8170 raft_consensus.cc:1679] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9caa1baaef84124a4a530b35945fad3 in term 58. I0825 15:49:49.709880 7904 leader_election.cc:347] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 58 election: Vote granted by peer 401cb2671e94452f87838515c7408bdc I0825 15:49:49.709902 7904 leader_election.cc:248] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. I0825 15:49:49.709930 8090 raft_consensus.cc:1679] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9caa1baaef84124a4a530b35945fad3 in term 58. I0825 15:49:49.710067 8349 raft_consensus.cc:1973] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 58 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.535s I0825 15:49:49.710069 7904 leader_election.cc:347] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 58 election: Vote granted by peer 92f34d06b19146db82cf42701d2363c5 I0825 15:49:49.710096 8349 raft_consensus.cc:1863] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 58 FOLLOWER]: Leader election won for term 58 I0825 15:49:49.710322 8349 raft_consensus.cc:468] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 58 LEADER]: Becoming Leader. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: LEADER Watermarks: {Received: term: 57 index: 2 Committed: term: 57 index: 2} I0825 15:49:49.710372 8349 consensus_queue.cc:145] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 57.2, Committed index: 57.2, Last appended: 57.2, Current term: 58, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } W0825 15:49:49.711109 8353 raft_consensus_state.cc:543] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 58 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 57 index: 2, New majority replicated is: term: 57 index: 2, Current term is: 58 I0825 15:49:49.711241 7902 catalog_manager.cc:2018] T 1f7a6bd64c604f7fada909bd575708cd reported consensus state change: term changed from 57 to 58, leader changed from 92f34d06b19146db82cf42701d2363c5 (127.108.26.1) to d9caa1baaef84124a4a530b35945fad3 (127.108.26.0). New consensus state: current_term: 58 leader_uuid: "d9caa1baaef84124a4a530b35945fad3" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } } I0825 15:49:50.084664 8090 raft_consensus.cc:873] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Refusing update from remote peer 401cb2671e94452f87838515c7408bdc: Log matching property violated. Preceding OpId in replica: term: 49 index: 2. Preceding OpId from leader: term: 50 index: 3. (index mismatch) I0825 15:49:50.084766 8151 raft_consensus.cc:873] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Refusing update from remote peer 401cb2671e94452f87838515c7408bdc: Log matching property violated. Preceding OpId in replica: term: 49 index: 2. Preceding OpId from leader: term: 50 index: 3. (index mismatch) I0825 15:49:50.084906 8354 consensus_queue.cc:590] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [LEADER]: Connected to new peer: Peer: 92f34d06b19146db82cf42701d2363c5, Is new: false, Last received: 49.2, Next index: 3, Last known committed idx: 2, Last exchange result: ERROR, Needs tablet copy: false I0825 15:49:50.085001 8330 consensus_queue.cc:590] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [LEADER]: Connected to new peer: Peer: d9caa1baaef84124a4a530b35945fad3, Is new: false, Last received: 49.2, Next index: 3, Last known committed idx: 2, Last exchange result: ERROR, Needs tablet copy: false I0825 15:49:50.086093 8357 raft_consensus_state.cc:536] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 50 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 49 index: 2 New committed index is: term: 50 index: 3 I0825 15:49:50.119021 8151 raft_consensus.cc:873] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 51 FOLLOWER]: Refusing update from remote peer 92f34d06b19146db82cf42701d2363c5: Log matching property violated. Preceding OpId in replica: term: 50 index: 2. Preceding OpId from leader: term: 51 index: 3. (index mismatch) I0825 15:49:50.119232 8358 consensus_queue.cc:590] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Connected to new peer: Peer: d9caa1baaef84124a4a530b35945fad3, Is new: false, Last received: 50.2, Next index: 3, Last known committed idx: 2, Last exchange result: ERROR, Needs tablet copy: false I0825 15:49:50.119253 8170 raft_consensus.cc:873] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 51 FOLLOWER]: Refusing update from remote peer 92f34d06b19146db82cf42701d2363c5: Log matching property violated. Preceding OpId in replica: term: 50 index: 2. Preceding OpId from leader: term: 51 index: 3. (index mismatch) I0825 15:49:50.119463 8340 consensus_queue.cc:590] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Connected to new peer: Peer: 401cb2671e94452f87838515c7408bdc, Is new: false, Last received: 50.2, Next index: 3, Last known committed idx: 2, Last exchange result: ERROR, Needs tablet copy: false I0825 15:49:50.120162 8361 raft_consensus_state.cc:536] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 51 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 50 index: 2 New committed index is: term: 51 index: 3 I0825 15:49:50.211328 8170 raft_consensus.cc:873] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 58 FOLLOWER]: Refusing update from remote peer d9caa1baaef84124a4a530b35945fad3: Log matching property violated. Preceding OpId in replica: term: 57 index: 2. Preceding OpId from leader: term: 58 index: 3. (index mismatch) I0825 15:49:50.211375 8090 raft_consensus.cc:873] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 58 FOLLOWER]: Refusing update from remote peer d9caa1baaef84124a4a530b35945fad3: Log matching property violated. Preceding OpId in replica: term: 57 index: 2. Preceding OpId from leader: term: 58 index: 3. (index mismatch) I0825 15:49:50.211593 8362 consensus_queue.cc:590] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Connected to new peer: Peer: 401cb2671e94452f87838515c7408bdc, Is new: false, Last received: 57.2, Next index: 3, Last known committed idx: 2, Last exchange result: ERROR, Needs tablet copy: false I0825 15:49:50.211625 8349 consensus_queue.cc:590] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Connected to new peer: Peer: 92f34d06b19146db82cf42701d2363c5, Is new: false, Last received: 57.2, Next index: 3, Last known committed idx: 2, Last exchange result: ERROR, Needs tablet copy: false I0825 15:49:50.212695 8365 raft_consensus_state.cc:536] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 58 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 57 index: 2 New committed index is: term: 58 index: 3 I0825 15:51:48.242246 7868 delta_tracker.cc:450] Flushing 36 deltas from DMS 1... I0825 15:51:48.243278 7868 delta_tracker.cc:402] Flushed delta block: 3974257638276197288 ts range: [6029989429292994560, 6029989439713271808] I0825 15:51:48.243321 7868 delta_tracker.cc:409] Reopened delta block for read: 3974257638276197288 I0825 15:51:48.244725 7868 maintenance_manager.cc:353] Time spent running FlushDeltaMemStoresOp(00000000000000000000000000000000): real 0.003s user 0.001s sys 0.000s