[dinesh@ve0518 debug]$ ./start-master.sh && ./start-tserver0.sh && ./start-tserver1.sh && ./start-tserver2.sh I0825 15:14:24.117295 6364 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0825 15:14:24.117410 6364 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0825 15:14:24.117377 6366 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0825 15:14:24.117465 6366 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0825 15:14:24.121318 6364 master_main.cc:58] Initializing master server... I0825 15:14:24.121325 6366 tablet_server_main.cc:54] Initializing tablet server... I0825 15:14:24.121762 6364 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 461841 I0825 15:14:24.122143 6366 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 461841 I0825 15:14:24.122738 6370 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0825 15:14:24.122738 6368 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0825 15:14:24.122891 6370 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0825 15:14:24.122894 6368 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0825 15:14:24.125963 6366 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0 uuid: "d9caa1baaef84124a4a530b35945fad3" format_stamp: "Formatted at 2016-08-25 21:52:23 on ve0518.halxg.cloudera.com" I0825 15:14:24.127140 6368 tablet_server_main.cc:54] Initializing tablet server... I0825 15:14:24.127140 6370 tablet_server_main.cc:54] Initializing tablet server... I0825 15:14:24.127604 6364 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0 uuid: "3b313b752ce844209ce84009c91c94df" format_stamp: "Formatted at 2016-08-25 21:52:23 on ve0518.halxg.cloudera.com" I0825 15:14:24.128062 6370 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 461841 I0825 15:14:24.128064 6368 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 461841 I0825 15:14:24.130220 6366 ts_tablet_manager.cc:609] Loading metadata for tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:14:24.131731 6366 ts_tablet_manager.cc:609] Loading metadata for tablet 17aec689be0047b987494787029f6e08 I0825 15:14:24.131948 6366 ts_tablet_manager.cc:609] Loading metadata for tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:14:24.132118 6368 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1 uuid: "92f34d06b19146db82cf42701d2363c5" format_stamp: "Formatted at 2016-08-25 21:52:23 on ve0518.halxg.cloudera.com" I0825 15:14:24.132118 6370 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2 uuid: "401cb2671e94452f87838515c7408bdc" format_stamp: "Formatted at 2016-08-25 21:52:23 on ve0518.halxg.cloudera.com" I0825 15:14:24.132344 6366 ts_tablet_manager.cc:765] Registered tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:14:24.132587 6366 ts_tablet_manager.cc:765] Registered tablet 17aec689be0047b987494787029f6e08 I0825 15:14:24.132638 6395 ts_tablet_manager.cc:633] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Bootstrapping tablet I0825 15:14:24.132642 6366 ts_tablet_manager.cc:765] Registered tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:14:24.132822 6366 tablet_server_main.cc:57] Starting tablet server... I0825 15:14:24.133164 6395 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Bootstrap starting. I0825 15:14:24.133496 6364 master_main.cc:61] Starting Master server... I0825 15:14:24.135030 6395 tablet_bootstrap.cc:554] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0825 15:14:24.135228 6395 tablet_bootstrap.cc:610] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 I0825 15:14:24.135248 6395 tablet_bootstrap.cc:618] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery in preparation for log replay W0825 15:14:24.135803 6395 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.135830 6395 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.135982 6395 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 for valid entry headers following offset 142... I0825 15:14:24.136854 6370 ts_tablet_manager.cc:609] Loading metadata for tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:14:24.136862 6368 ts_tablet_manager.cc:609] Loading metadata for tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:14:24.137724 6364 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:57007 I0825 15:14:24.138031 6364 webserver.cc:122] Starting webserver on localhost:0 I0825 15:14:24.138046 6364 webserver.cc:133] Document root disabled I0825 15:14:24.138447 6364 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0825 15:14:24.138464 6368 ts_tablet_manager.cc:609] Loading metadata for tablet 17aec689be0047b987494787029f6e08 I0825 15:14:24.138465 6370 ts_tablet_manager.cc:609] Loading metadata for tablet 17aec689be0047b987494787029f6e08 I0825 15:14:24.138736 6370 ts_tablet_manager.cc:609] Loading metadata for tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:14:24.138737 6368 ts_tablet_manager.cc:609] Loading metadata for tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:14:24.138751 6364 webserver.cc:216] Webserver started. Bound to: http://127.0.0.1:52504/ I0825 15:14:24.139219 6370 ts_tablet_manager.cc:765] Registered tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:14:24.139219 6368 ts_tablet_manager.cc:765] Registered tablet f32f6e0b1f354643b6b030599a359fa6 I0825 15:14:24.139269 6364 server_base.cc:234] Dumped server information to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/info.pb I0825 15:14:24.139502 6370 ts_tablet_manager.cc:765] Registered tablet 17aec689be0047b987494787029f6e08 I0825 15:14:24.139508 6512 ts_tablet_manager.cc:633] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Bootstrapping tablet I0825 15:14:24.139549 6368 ts_tablet_manager.cc:765] Registered tablet 17aec689be0047b987494787029f6e08 I0825 15:14:24.139583 6513 ts_tablet_manager.cc:633] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Bootstrapping tablet I0825 15:14:24.139614 6368 ts_tablet_manager.cc:765] Registered tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:14:24.139629 6370 ts_tablet_manager.cc:765] Registered tablet 1f7a6bd64c604f7fada909bd575708cd I0825 15:14:24.139761 6368 tablet_server_main.cc:57] Starting tablet server... I0825 15:14:24.139818 6370 tablet_server_main.cc:57] Starting tablet server... I0825 15:14:24.139924 6512 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Bootstrap starting. I0825 15:14:24.140028 6513 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Bootstrap starting. I0825 15:14:24.141216 6512 tablet_bootstrap.cc:554] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Time spent opening tablet: real 0.000s user 0.001s sys 0.000s I0825 15:14:24.141379 6512 tablet_bootstrap.cc:610] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 I0825 15:14:24.141398 6512 tablet_bootstrap.cc:618] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery in preparation for log replay I0825 15:14:24.141445 6513 tablet_bootstrap.cc:554] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0825 15:14:24.141609 6513 tablet_bootstrap.cc:610] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 I0825 15:14:24.141629 6513 tablet_bootstrap.cc:618] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery in preparation for log replay W0825 15:14:24.141856 6512 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.141885 6512 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. W0825 15:14:24.142097 6513 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.142120 6513 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.142148 6516 tablet_bootstrap.cc:394] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Bootstrap starting. I0825 15:14:24.142221 6513 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 for valid entry headers following offset 142... I0825 15:14:24.142807 6366 rpc_server.cc:164] RPC server started. Bound to: 127.108.26.0:37291 I0825 15:14:24.143075 6366 webserver.cc:122] Starting webserver on 127.108.26.0:0 I0825 15:14:24.143090 6366 webserver.cc:133] Document root disabled I0825 15:14:24.143419 6366 webserver.cc:216] Webserver started. Bound to: http://127.108.26.0:38131/ I0825 15:14:24.143646 6512 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 for valid entry headers following offset 180... I0825 15:14:24.143822 6366 server_base.cc:234] Dumped server information to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/info.pb I0825 15:14:24.144174 6366 tablet_server_main.cc:60] Tablet server successfully started. I0825 15:14:24.145365 6516 tablet_bootstrap.cc:554] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Time spent opening tablet: real 0.002s user 0.002s sys 0.000s I0825 15:14:24.145500 6516 tablet_bootstrap.cc:610] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 I0825 15:14:24.145514 6516 tablet_bootstrap.cc:618] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery in preparation for log replay W0825 15:14:24.145854 6516 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.145875 6516 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.147984 6620 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:57007 I0825 15:14:24.148020 6516 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 for valid entry headers following offset 24088... I0825 15:14:24.148069 6620 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.148126 6620 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.149266 6620 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.149307 6620 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.149334 6620 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.149842 6620 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.149880 6620 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.149907 6620 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.150259 6368 rpc_server.cc:164] RPC server started. Bound to: 127.108.26.1:33958 W0825 15:14:24.150363 6620 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized W0825 15:14:24.150390 6620 heartbeater.cc:283] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0825 15:14:24.150427 6620 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.150454 6620 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.150573 6368 webserver.cc:122] Starting webserver on 127.108.26.1:0 I0825 15:14:24.150588 6368 webserver.cc:133] Document root disabled W0825 15:14:24.150876 6620 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.150943 6370 rpc_server.cc:164] RPC server started. Bound to: 127.108.26.2:35954 I0825 15:14:24.150984 6368 webserver.cc:216] Webserver started. Bound to: http://127.108.26.1:48742/ I0825 15:14:24.151160 6370 webserver.cc:122] Starting webserver on 127.108.26.2:0 I0825 15:14:24.151186 6370 webserver.cc:133] Document root disabled I0825 15:14:24.151481 6368 server_base.cc:234] Dumped server information to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/info.pb I0825 15:14:24.151528 6370 webserver.cc:216] Webserver started. Bound to: http://127.108.26.2:49926/ I0825 15:14:24.151774 6368 tablet_server_main.cc:60] Tablet server successfully started. I0825 15:14:24.151903 6370 server_base.cc:234] Dumped server information to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/info.pb I0825 15:14:24.152196 6370 tablet_server_main.cc:60] Tablet server successfully started. I0825 15:14:24.154254 6765 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:57007 I0825 15:14:24.154325 6765 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.154372 6765 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.155015 6767 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:57007 W0825 15:14:24.155056 6765 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.155084 6765 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.155103 6765 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.155108 6767 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.155170 6767 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.155504 6765 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.155529 6765 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.155549 6765 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.155850 6767 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.155882 6767 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.155905 6767 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.156008 6765 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized W0825 15:14:24.156023 6765 heartbeater.cc:283] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0825 15:14:24.156052 6765 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.156072 6765 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.156319 6767 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.156347 6767 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.156368 6767 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.156409 6765 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized W0825 15:14:24.156719 6767 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized W0825 15:14:24.156731 6767 heartbeater.cc:283] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0825 15:14:24.156757 6767 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.156792 6767 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.157196 6767 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.191339 6513 log_util.cc:606] Found no log entry headers I0825 15:14:24.191387 6513 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 130 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Prior entries: I0825 15:14:24.191450 6513 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 (valid entries through byte offset 130) I0825 15:14:24.191980 6513 tablet.cc:855] T f32f6e0b1f354643b6b030599a359fa6 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.192996 6513 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0825 15:14:24.193580 6395 log_util.cc:606] Found no log entry headers I0825 15:14:24.193625 6395 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 130 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Prior entries: I0825 15:14:24.193686 6395 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 (valid entries through byte offset 130) I0825 15:14:24.194211 6395 tablet.cc:855] T f32f6e0b1f354643b6b030599a359fa6 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.194268 6513 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:14:24.194310 6513 tablet_bootstrap.cc:1031] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: ReplayState: Previous OpId: 0.0, Committed OpId: 0.0, Pending Replicates: 0, Pending Commits: 0 I0825 15:14:24.195221 6395 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0825 15:14:24.195482 6513 tablet_bootstrap.cc:647] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery I0825 15:14:24.195502 6513 tablet_bootstrap.cc:650] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264195497 I0825 15:14:24.195525 6513 tablet_bootstrap.cc:660] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264195497 I0825 15:14:24.195643 6513 tablet_bootstrap.cc:663] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264195497 I0825 15:14:24.195703 6513 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Bootstrap complete. I0825 15:14:24.195930 6516 log_util.cc:606] Found no log entry headers I0825 15:14:24.195977 6516 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 24076 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001: Prior entries: [off=23466 REPLICATE (0.41)] [off=23500 COMMIT (0.41)] [off=24042 REPLICATE (0.42)] [off=24076 COMMIT (0.42)] I0825 15:14:24.196007 6516 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery/wal-000000001 (valid entries through byte offset 24076) I0825 15:14:24.196295 6395 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:14:24.196310 6516 tablet.cc:855] T 00000000000000000000000000000000 Rewinding schema during bootstrap to Schema [ 10:entry_type[int8 NOT NULL], 11:entry_id[string NOT NULL], 12:metadata[string NOT NULL] ] I0825 15:14:24.196341 6395 tablet_bootstrap.cc:1031] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: ReplayState: Previous OpId: 0.0, Committed OpId: 0.0, Pending Replicates: 0, Pending Commits: 0 I0825 15:14:24.196992 6516 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0825 15:14:24.197530 6395 tablet_bootstrap.cc:647] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery I0825 15:14:24.197552 6395 tablet_bootstrap.cc:650] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264197546 I0825 15:14:24.197579 6395 tablet_bootstrap.cc:660] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264197546 I0825 15:14:24.197736 6395 tablet_bootstrap.cc:663] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264197546 I0825 15:14:24.197790 6395 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Bootstrap complete. I0825 15:14:24.199162 6512 log_util.cc:606] Found no log entry headers I0825 15:14:24.199184 6512 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 168 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001: Prior entries: [off=168 REPLICATE (1.1)] I0825 15:14:24.199216 6512 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery/wal-000000001 (valid entries through byte offset 168) I0825 15:14:24.199498 6512 tablet.cc:855] T f32f6e0b1f354643b6b030599a359fa6 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.200286 6512 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0825 15:14:24.201587 6512 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0825 15:14:24.201730 6512 tablet_bootstrap.cc:1031] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: ReplayState: Previous OpId: 1.1, Committed OpId: 0.0, Pending Replicates: 1, Pending Commits: 0 I0825 15:14:24.201738 6512 tablet_bootstrap.cc:1031] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Dumping REPLICATES: I0825 15:14:24.201742 6512 tablet_bootstrap.cc:1031] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: type: REPLICATE replicate { id { term: 1 index: 1 } timestamp: 6029975321926193152 op_type: NO_OP noop_request { } } I0825 15:14:24.202376 6512 tablet_bootstrap.cc:647] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery I0825 15:14:24.202392 6512 tablet_bootstrap.cc:650] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264202388 I0825 15:14:24.202414 6512 tablet_bootstrap.cc:660] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264202388 I0825 15:14:24.202577 6512 tablet_bootstrap.cc:663] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/f32f6e0b1f354643b6b030599a359fa6.recovery-1472163264202388 I0825 15:14:24.202610 6512 tablet_bootstrap.cc:394] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Bootstrap complete. I0825 15:14:24.202867 6513 ts_tablet_manager.cc:638] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Time spent bootstrapping tablet: real 0.063s user 0.025s sys 0.037s I0825 15:14:24.204181 6513 raft_consensus.cc:269] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 43 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:24.204468 6513 raft_consensus.cc:478] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 43 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401cb2671e94452f87838515c7408bdc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0825 15:14:24.204540 6513 consensus_queue.cc:162] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.204900 6767 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.204958 6767 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.205130 6513 ts_tablet_manager.cc:664] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I0825 15:14:24.205229 6513 ts_tablet_manager.cc:633] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Bootstrapping tablet I0825 15:14:24.205368 6513 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Bootstrap starting. I0825 15:14:24.205910 6513 tablet_bootstrap.cc:554] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:14:24.205987 6513 tablet_bootstrap.cc:610] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 I0825 15:14:24.205998 6513 tablet_bootstrap.cc:618] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery in preparation for log replay W0825 15:14:24.206004 6767 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.206037 6767 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.206058 6767 heartbeater.cc:325] Sending a full tablet report to master... W0825 15:14:24.206152 6513 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.206168 6513 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.206188 6513 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 for valid entry headers following offset 142... W0825 15:14:24.206575 6767 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.206985 6516 tablet_bootstrap.cc:394] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=41} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:14:24.207020 6516 tablet_bootstrap.cc:1031] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: ReplayState: Previous OpId: 0.42, Committed OpId: 0.42, Pending Replicates: 0, Pending Commits: 0 I0825 15:14:24.207520 6516 tablet_bootstrap.cc:647] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery I0825 15:14:24.207535 6516 tablet_bootstrap.cc:650] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery-1472163264207531 I0825 15:14:24.207562 6516 tablet_bootstrap.cc:660] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery-1472163264207531 I0825 15:14:24.207765 6516 tablet_bootstrap.cc:663] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/master-0/wals/00000000000000000000000000000000.recovery-1472163264207531 I0825 15:14:24.207795 6516 tablet_bootstrap.cc:394] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Bootstrap complete. I0825 15:14:24.208515 6395 ts_tablet_manager.cc:638] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Time spent bootstrapping tablet: real 0.076s user 0.031s sys 0.043s I0825 15:14:24.209933 6395 raft_consensus.cc:269] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 48 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:24.210237 6395 raft_consensus.cc:478] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 48 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0825 15:14:24.210321 6395 consensus_queue.cc:162] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.210728 6620 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.210790 6620 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.210937 6395 ts_tablet_manager.cc:664] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s I0825 15:14:24.211038 6395 ts_tablet_manager.cc:633] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Bootstrapping tablet I0825 15:14:24.211220 6395 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Bootstrap starting. W0825 15:14:24.211500 6620 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.211544 6620 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.211570 6620 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.211817 6395 tablet_bootstrap.cc:554] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Time spent opening tablet: real 0.000s user 0.001s sys 0.000s I0825 15:14:24.211916 6395 tablet_bootstrap.cc:610] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 I0825 15:14:24.211930 6395 tablet_bootstrap.cc:618] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery in preparation for log replay W0825 15:14:24.212075 6620 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized W0825 15:14:24.212121 6395 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.212136 6395 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.212927 6512 ts_tablet_manager.cc:638] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Time spent bootstrapping tablet: real 0.073s user 0.024s sys 0.049s I0825 15:14:24.213248 6395 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 for valid entry headers following offset 180... I0825 15:14:24.213913 6512 raft_consensus.cc:269] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 47 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:24.214202 6512 raft_consensus.cc:478] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 47 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 1 Committed: term: 0 index: 0} I0825 15:14:24.214258 6512 consensus_queue.cc:162] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.214555 6765 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.214607 6765 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.214722 6512 ts_tablet_manager.cc:664] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0825 15:14:24.214812 6512 ts_tablet_manager.cc:633] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Bootstrapping tablet I0825 15:14:24.214951 6512 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Bootstrap starting. W0825 15:14:24.215350 6765 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.215370 6765 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.215394 6765 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.215432 6516 local_consensus.cc:60] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df: Starting LocalConsensus... I0825 15:14:24.215457 6512 tablet_bootstrap.cc:554] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:14:24.215529 6512 tablet_bootstrap.cc:610] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 I0825 15:14:24.215540 6512 tablet_bootstrap.cc:618] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08 to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery in preparation for log replay I0825 15:14:24.215574 6516 sys_catalog.cc:237] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "3b313b752ce844209ce84009c91c94df" config { opid_index: -1 local: true peers { permanent_uuid: "3b313b752ce844209ce84009c91c94df" member_type: VOTER } } I0825 15:14:24.215595 6516 sys_catalog.cc:240] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER W0825 15:14:24.215670 6512 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.215682 6512 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.215724 6512 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 for valid entry headers following offset 142... W0825 15:14:24.215989 6765 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:57007: Service unavailable: catalog manager has not been initialized I0825 15:14:24.216183 6516 sys_catalog.cc:314] T 00000000000000000000000000000000 P 3b313b752ce844209ce84009c91c94df [sys.catalog]: configured and running, proceeding with master startup. I0825 15:14:24.216188 6786 catalog_manager.cc:627] Loading table and tablet metadata into memory... I0825 15:14:24.220155 6786 catalog_manager.cc:211] Loaded metadata for table test-table [id=162312d1479340109aec6ff77ce314f1] I0825 15:14:24.221405 6786 catalog_manager.cc:261] Loaded metadata for tablet 17aec689be0047b987494787029f6e08 (table test-table [id=162312d1479340109aec6ff77ce314f1]) I0825 15:14:24.221565 6786 catalog_manager.cc:261] Loaded metadata for tablet 1f7a6bd64c604f7fada909bd575708cd (table test-table [id=162312d1479340109aec6ff77ce314f1]) I0825 15:14:24.221720 6786 catalog_manager.cc:261] Loaded metadata for tablet f32f6e0b1f354643b6b030599a359fa6 (table test-table [id=162312d1479340109aec6ff77ce314f1]) W0825 15:14:24.221964 6788 catalog_manager.cc:348] Catalog manager is not initialized! I0825 15:14:24.222079 6364 master_main.cc:64] Master server successfully started. I0825 15:14:24.252359 6513 log_util.cc:606] Found no log entry headers I0825 15:14:24.252400 6513 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 130 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Prior entries: I0825 15:14:24.252424 6513 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 (valid entries through byte offset 130) I0825 15:14:24.252689 6513 tablet.cc:855] T 17aec689be0047b987494787029f6e08 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.254199 6513 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:14:24.254230 6513 tablet_bootstrap.cc:1031] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: ReplayState: Previous OpId: 0.0, Committed OpId: 0.0, Pending Replicates: 0, Pending Commits: 0 I0825 15:14:24.254564 6513 tablet_bootstrap.cc:647] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery I0825 15:14:24.254577 6513 tablet_bootstrap.cc:650] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery-1472163264254575 I0825 15:14:24.254597 6513 tablet_bootstrap.cc:660] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery-1472163264254575 I0825 15:14:24.254659 6513 tablet_bootstrap.cc:663] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/17aec689be0047b987494787029f6e08.recovery-1472163264254575 I0825 15:14:24.254681 6513 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Bootstrap complete. I0825 15:14:24.261416 6513 ts_tablet_manager.cc:638] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Time spent bootstrapping tablet: real 0.056s user 0.019s sys 0.037s I0825 15:14:24.262109 6513 raft_consensus.cc:269] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 47 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:24.262163 6513 raft_consensus.cc:478] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 47 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401cb2671e94452f87838515c7408bdc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0825 15:14:24.262197 6513 consensus_queue.cc:162] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.262411 6767 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.262470 6767 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.262491 6513 ts_tablet_manager.cc:664] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0825 15:14:24.262557 6513 ts_tablet_manager.cc:633] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Bootstrapping tablet I0825 15:14:24.262689 6513 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Bootstrap starting. I0825 15:14:24.263257 6513 tablet_bootstrap.cc:554] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:14:24.263334 6513 tablet_bootstrap.cc:610] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 I0825 15:14:24.263345 6513 tablet_bootstrap.cc:618] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery in preparation for log replay W0825 15:14:24.263480 6513 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.263492 6513 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.263517 6513 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 for valid entry headers following offset 142... I0825 15:14:24.263579 6433 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "401cb2671e94452f87838515c7408bdc" instance_seqno: 1472163264148703 } with Master I0825 15:14:24.264892 6395 log_util.cc:606] Found no log entry headers I0825 15:14:24.264925 6395 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 168 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Prior entries: [off=168 REPLICATE (1.1)] I0825 15:14:24.264958 6395 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 (valid entries through byte offset 168) I0825 15:14:24.265244 6395 tablet.cc:855] T 17aec689be0047b987494787029f6e08 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.266577 6512 log_util.cc:606] Found no log entry headers I0825 15:14:24.266594 6512 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 130 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001: Prior entries: I0825 15:14:24.266607 6512 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery/wal-000000001 (valid entries through byte offset 130) I0825 15:14:24.266789 6512 tablet.cc:855] T 17aec689be0047b987494787029f6e08 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.267033 6395 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0825 15:14:24.267139 6395 tablet_bootstrap.cc:1031] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: ReplayState: Previous OpId: 1.1, Committed OpId: 0.0, Pending Replicates: 1, Pending Commits: 0 I0825 15:14:24.267150 6395 tablet_bootstrap.cc:1031] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Dumping REPLICATES: I0825 15:14:24.267156 6395 tablet_bootstrap.cc:1031] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: type: REPLICATE replicate { id { term: 1 index: 1 } timestamp: 6029975321432735744 op_type: NO_OP noop_request { } } I0825 15:14:24.267536 6395 tablet_bootstrap.cc:647] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery I0825 15:14:24.267551 6395 tablet_bootstrap.cc:650] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery-1472163264267549 I0825 15:14:24.267575 6395 tablet_bootstrap.cc:660] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery-1472163264267549 I0825 15:14:24.267710 6395 tablet_bootstrap.cc:663] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/17aec689be0047b987494787029f6e08.recovery-1472163264267549 I0825 15:14:24.267736 6395 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Bootstrap complete. I0825 15:14:24.267848 6512 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:14:24.267876 6512 tablet_bootstrap.cc:1031] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: ReplayState: Previous OpId: 0.0, Committed OpId: 0.0, Pending Replicates: 0, Pending Commits: 0 I0825 15:14:24.268146 6512 tablet_bootstrap.cc:647] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery I0825 15:14:24.268158 6512 tablet_bootstrap.cc:650] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery-1472163264268157 I0825 15:14:24.268178 6512 tablet_bootstrap.cc:660] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery-1472163264268157 I0825 15:14:24.268232 6512 tablet_bootstrap.cc:663] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/17aec689be0047b987494787029f6e08.recovery-1472163264268157 I0825 15:14:24.268247 6512 tablet_bootstrap.cc:394] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Bootstrap complete. I0825 15:14:24.277791 6395 ts_tablet_manager.cc:638] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Time spent bootstrapping tablet: real 0.067s user 0.021s sys 0.046s I0825 15:14:24.278144 6512 ts_tablet_manager.cc:638] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Time spent bootstrapping tablet: real 0.063s user 0.023s sys 0.041s I0825 15:14:24.278509 6395 raft_consensus.cc:269] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 46 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:24.278692 6395 raft_consensus.cc:478] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 46 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 1 Committed: term: 0 index: 0} I0825 15:14:24.278702 6512 raft_consensus.cc:269] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 48 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:24.278729 6395 consensus_queue.cc:162] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.278746 6512 raft_consensus.cc:478] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 48 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0825 15:14:24.278796 6512 consensus_queue.cc:162] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.278988 6620 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.279014 6765 heartbeater.cc:319] Registering TS with master... I0825 15:14:24.279042 6395 ts_tablet_manager.cc:664] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0825 15:14:24.279047 6512 ts_tablet_manager.cc:664] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0825 15:14:24.279052 6620 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.279072 6765 heartbeater.cc:325] Sending a full tablet report to master... I0825 15:14:24.279175 6512 ts_tablet_manager.cc:633] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Bootstrapping tablet I0825 15:14:24.279184 6395 ts_tablet_manager.cc:633] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Bootstrapping tablet I0825 15:14:24.279314 6512 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Bootstrap starting. I0825 15:14:24.279357 6395 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Bootstrap starting. I0825 15:14:24.279662 6433 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" instance_seqno: 1472163264140568 } with Master I0825 15:14:24.279737 6432 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" instance_seqno: 1472163264148159 } with Master I0825 15:14:24.279742 6512 tablet_bootstrap.cc:554] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0825 15:14:24.279837 6512 tablet_bootstrap.cc:610] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 I0825 15:14:24.279849 6512 tablet_bootstrap.cc:618] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery in preparation for log replay I0825 15:14:24.279901 6395 tablet_bootstrap.cc:554] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s W0825 15:14:24.279976 6512 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.279978 6395 tablet_bootstrap.cc:610] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Will attempt to recover log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd/wal-000000001 to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 I0825 15:14:24.279988 6512 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.279991 6395 tablet_bootstrap.cc:618] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Moving log directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd to recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery in preparation for log replay I0825 15:14:24.280035 6512 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 for valid entry headers following offset 180... W0825 15:14:24.280135 6395 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0825 15:14:24.280148 6395 log_reader.cc:152] Log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0825 15:14:24.280172 6395 log_util.cc:569] Scanning /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 for valid entry headers following offset 142... I0825 15:14:24.309275 6513 log_util.cc:606] Found no log entry headers I0825 15:14:24.309305 6513 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 130 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Prior entries: I0825 15:14:24.309329 6513 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 (valid entries through byte offset 130) I0825 15:14:24.309568 6513 tablet.cc:855] T 1f7a6bd64c604f7fada909bd575708cd Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.311024 6513 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:14:24.311079 6513 tablet_bootstrap.cc:1031] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: ReplayState: Previous OpId: 0.0, Committed OpId: 0.0, Pending Replicates: 0, Pending Commits: 0 I0825 15:14:24.311532 6513 tablet_bootstrap.cc:647] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery I0825 15:14:24.311553 6513 tablet_bootstrap.cc:650] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264311548 I0825 15:14:24.311583 6513 tablet_bootstrap.cc:660] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264311548 I0825 15:14:24.311687 6513 tablet_bootstrap.cc:663] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-2/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264311548 I0825 15:14:24.311738 6513 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Bootstrap complete. I0825 15:14:24.319461 6513 ts_tablet_manager.cc:638] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Time spent bootstrapping tablet: real 0.057s user 0.020s sys 0.037s I0825 15:14:24.320245 6513 raft_consensus.cc:269] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 46 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:14:24.320300 6513 raft_consensus.cc:478] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 46 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401cb2671e94452f87838515c7408bdc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0825 15:14:24.320334 6513 consensus_queue.cc:162] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.320698 6513 ts_tablet_manager.cc:664] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0825 15:14:24.328280 6512 log_util.cc:606] Found no log entry headers I0825 15:14:24.328315 6512 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 168 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Prior entries: [off=168 REPLICATE (1.1)] I0825 15:14:24.328331 6512 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 (valid entries through byte offset 168) I0825 15:14:24.328495 6512 tablet.cc:855] T 1f7a6bd64c604f7fada909bd575708cd Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.329946 6512 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0825 15:14:24.330008 6512 tablet_bootstrap.cc:1031] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: ReplayState: Previous OpId: 1.1, Committed OpId: 0.0, Pending Replicates: 1, Pending Commits: 0 I0825 15:14:24.330014 6512 tablet_bootstrap.cc:1031] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Dumping REPLICATES: I0825 15:14:24.330018 6512 tablet_bootstrap.cc:1031] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: type: REPLICATE replicate { id { term: 1 index: 1 } timestamp: 6029975321384198144 op_type: NO_OP noop_request { } } I0825 15:14:24.330126 6395 log_util.cc:606] Found no log entry headers I0825 15:14:24.330149 6395 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 130 for log segment /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001: Prior entries: I0825 15:14:24.330165 6395 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery/wal-000000001 (valid entries through byte offset 130) I0825 15:14:24.330291 6512 tablet_bootstrap.cc:647] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery I0825 15:14:24.330303 6512 tablet_bootstrap.cc:650] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264330301 I0825 15:14:24.330322 6512 tablet_bootstrap.cc:660] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264330301 I0825 15:14:24.330338 6395 tablet.cc:855] T 1f7a6bd64c604f7fada909bd575708cd Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0825 15:14:24.330428 6512 tablet_bootstrap.cc:663] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-1/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264330301 I0825 15:14:24.330446 6512 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Bootstrap complete. I0825 15:14:24.331507 6395 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0825 15:14:24.331537 6395 tablet_bootstrap.cc:1031] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: ReplayState: Previous OpId: 0.0, Committed OpId: 0.0, Pending Replicates: 0, Pending Commits: 0 I0825 15:14:24.331848 6395 tablet_bootstrap.cc:647] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Preparing to delete log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery I0825 15:14:24.331864 6395 tablet_bootstrap.cc:650] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Renaming log recovery dir from /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery to /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264331861 I0825 15:14:24.331887 6395 tablet_bootstrap.cc:660] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Deleting all files from renamed log recovery directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264331861 I0825 15:14:24.331940 6395 tablet_bootstrap.cc:663] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Completed deletion of old log recovery files and directory /tmp/kudutest-4519/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1472161943177725-27674/minicluster-data/ts-0/wals/1f7a6bd64c604f7fada909bd575708cd.recovery-1472163264331861 I0825 15:14:24.331959 6395 tablet_bootstrap.cc:394] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Bootstrap complete. I0825 15:14:24.340060 6512 ts_tablet_manager.cc:638] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Time spent bootstrapping tablet: real 0.061s user 0.022s sys 0.040s I0825 15:14:24.340643 6512 raft_consensus.cc:269] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 46 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:14:24.340708 6512 raft_consensus.cc:478] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 46 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 1 Committed: term: 0 index: 0} I0825 15:14:24.340739 6512 consensus_queue.cc:162] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.341087 6512 ts_tablet_manager.cc:664] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0825 15:14:24.341379 6395 ts_tablet_manager.cc:638] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Time spent bootstrapping tablet: real 0.062s user 0.020s sys 0.041s I0825 15:14:24.342046 6395 raft_consensus.cc:269] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 55 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:14:24.342108 6395 raft_consensus.cc:478] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 55 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0825 15:14:24.342138 6395 consensus_queue.cc:162] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:24.342425 6395 ts_tablet_manager.cc:664] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s [dinesh@ve0518 debug]$ I0825 15:14:25.829818 6791 raft_consensus.cc:363] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 47 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:14:25.829849 6791 raft_consensus.cc:1980] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 47 FOLLOWER]: Advancing to term 48 I0825 15:14:25.830371 6791 raft_consensus.cc:1931] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 48 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.868s I0825 15:14:25.830427 6791 raft_consensus.cc:379] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 48 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:25.831483 6791 leader_election.cc:222] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 48 election: Requesting vote from peer 92f34d06b19146db82cf42701d2363c5 I0825 15:14:25.831634 6791 leader_election.cc:222] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 48 election: Requesting vote from peer d9caa1baaef84124a4a530b35945fad3 I0825 15:14:25.833935 6506 raft_consensus.cc:1980] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 46 FOLLOWER]: Advancing to term 48 I0825 15:14:25.834007 6674 raft_consensus.cc:1572] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 401cb2671e94452f87838515c7408bdc in current term 48: Already voted for candidate 92f34d06b19146db82cf42701d2363c5 in this term. I0825 15:14:25.834427 6506 raft_consensus.cc:1589] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 401cb2671e94452f87838515c7408bdc for term 48 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0825 15:14:25.834640 6444 leader_election.cc:360] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 48 election: Vote denied by peer 92f34d06b19146db82cf42701d2363c5. Message: Invalid argument: T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 401cb2671e94452f87838515c7408bdc in current term 48: Already voted for candidate 92f34d06b19146db82cf42701d2363c5 in this term. I0825 15:14:25.834959 6437 leader_election.cc:360] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 48 election: Vote denied by peer d9caa1baaef84124a4a530b35945fad3. Message: Invalid argument: T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 401cb2671e94452f87838515c7408bdc for term 48 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0825 15:14:25.835006 6437 leader_election.cc:247] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. I0825 15:14:25.835373 6820 raft_consensus.cc:1931] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 48 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.722s I0825 15:14:25.835444 6820 raft_consensus.cc:1784] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 48 FOLLOWER]: Leader election lost for term 48. Reason: None given I0825 15:14:25.870713 6814 raft_consensus.cc:363] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 55 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:14:25.870731 6814 raft_consensus.cc:1980] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 55 FOLLOWER]: Advancing to term 56 I0825 15:14:25.870981 6814 raft_consensus.cc:1931] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.568s I0825 15:14:25.871023 6814 raft_consensus.cc:379] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:14:25.871816 6814 leader_election.cc:222] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 56 election: Requesting vote from peer 401cb2671e94452f87838515c7408bdc I0825 15:14:25.871912 6814 leader_election.cc:222] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 56 election: Requesting vote from peer 92f34d06b19146db82cf42701d2363c5 I0825 15:14:25.873383 6674 raft_consensus.cc:1980] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 46 FOLLOWER]: Advancing to term 56 I0825 15:14:25.873584 6683 raft_consensus.cc:1980] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 46 FOLLOWER]: Advancing to term 56 I0825 15:14:25.873838 6674 raft_consensus.cc:1589] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate d9caa1baaef84124a4a530b35945fad3 for term 56 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0825 15:14:25.874001 6683 raft_consensus.cc:1931] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 56 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.213s I0825 15:14:25.874121 6383 leader_election.cc:360] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 56 election: Vote denied by peer 92f34d06b19146db82cf42701d2363c5. Message: Invalid argument: T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate d9caa1baaef84124a4a530b35945fad3 for term 56 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0825 15:14:25.874414 6683 raft_consensus.cc:1640] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9caa1baaef84124a4a530b35945fad3 in term 56. I0825 15:14:25.874856 6383 leader_election.cc:346] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 56 election: Vote granted by peer 401cb2671e94452f87838515c7408bdc I0825 15:14:25.874886 6383 leader_election.cc:247] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. I0825 15:14:25.875171 6823 raft_consensus.cc:1931] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.883s I0825 15:14:25.875210 6823 raft_consensus.cc:1822] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 FOLLOWER]: Leader election won for term 56 I0825 15:14:25.875488 6823 raft_consensus.cc:443] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 LEADER]: Becoming Leader. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0825 15:14:25.875699 6823 consensus_queue.cc:145] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 56, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:14:25.877763 6433 catalog_manager.cc:1650] Tablet: 1f7a6bd64c604f7fada909bd575708cd reported consensus state change. New consensus state: current_term: 56 leader_uuid: "d9caa1baaef84124a4a530b35945fad3" config { opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } } W0825 15:14:25.877954 6827 raft_consensus_state.cc:541] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 56 I0825 15:14:25.922194 6812 raft_consensus.cc:363] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 56 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:14:25.922216 6812 raft_consensus.cc:1980] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 56 FOLLOWER]: Advancing to term 57 I0825 15:14:25.922569 6812 raft_consensus.cc:1931] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.837s I0825 15:14:25.922610 6812 raft_consensus.cc:379] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:14:25.923295 6812 leader_election.cc:222] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 57 election: Requesting vote from peer 401cb2671e94452f87838515c7408bdc I0825 15:14:25.923383 6812 leader_election.cc:222] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 57 election: Requesting vote from peer d9caa1baaef84124a4a530b35945fad3 I0825 15:14:25.924440 6683 raft_consensus.cc:1980] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 56 FOLLOWER]: Advancing to term 57 I0825 15:14:25.924535 6506 raft_consensus.cc:1603] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 LEADER]: Leader election vote request: Denying vote to candidate 92f34d06b19146db82cf42701d2363c5 for term 57 because replica is either leader or believes a valid leader to be alive. I0825 15:14:25.924746 6436 leader_election.cc:360] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 57 election: Vote denied by peer d9caa1baaef84124a4a530b35945fad3. Message: Invalid argument: T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 LEADER]: Leader election vote request: Denying vote to candidate 92f34d06b19146db82cf42701d2363c5 for term 57 because replica is either leader or believes a valid leader to be alive. I0825 15:14:25.924809 6683 raft_consensus.cc:1931] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 57 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.202s I0825 15:14:25.925168 6683 raft_consensus.cc:1640] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f34d06b19146db82cf42701d2363c5 in term 57. I0825 15:14:25.925370 6445 leader_election.cc:346] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 57 election: Vote granted by peer 401cb2671e94452f87838515c7408bdc I0825 15:14:25.925405 6445 leader_election.cc:247] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. I0825 15:14:25.925645 6831 raft_consensus.cc:1931] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.139s I0825 15:14:25.925681 6831 raft_consensus.cc:1822] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 FOLLOWER]: Leader election won for term 57 I0825 15:14:25.925983 6831 raft_consensus.cc:443] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 LEADER]: Becoming Leader. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 1 Committed: term: 0 index: 0} I0825 15:14:25.926172 6831 consensus_queue.cc:145] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 57, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } I0825 15:14:25.927829 6433 catalog_manager.cc:1650] Tablet: 1f7a6bd64c604f7fada909bd575708cd reported consensus state change. New consensus state: current_term: 57 leader_uuid: "92f34d06b19146db82cf42701d2363c5" config { opid_index: -1 local: false peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } } W0825 15:14:25.927851 6835 raft_consensus_state.cc:541] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 57 I0825 15:14:26.068382 6782 raft_consensus.cc:363] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 48 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:14:26.068406 6782 raft_consensus.cc:1980] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 48 FOLLOWER]: Advancing to term 49 I0825 15:14:26.068771 6782 raft_consensus.cc:1931] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 18.745s I0825 15:14:26.068816 6782 raft_consensus.cc:379] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:26.069419 6782 leader_election.cc:222] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Requesting vote from peer 92f34d06b19146db82cf42701d2363c5 I0825 15:14:26.069499 6782 leader_election.cc:222] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Requesting vote from peer 401cb2671e94452f87838515c7408bdc I0825 15:14:26.069799 6683 raft_consensus.cc:1980] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 43 FOLLOWER]: Advancing to term 49 I0825 15:14:26.069871 6674 raft_consensus.cc:1980] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 47 FOLLOWER]: Advancing to term 49 I0825 15:14:26.070139 6683 raft_consensus.cc:1931] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.919s I0825 15:14:26.070217 6674 raft_consensus.cc:1589] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate d9caa1baaef84124a4a530b35945fad3 for term 49 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0825 15:14:26.070452 6383 leader_election.cc:360] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Vote denied by peer 92f34d06b19146db82cf42701d2363c5. Message: Invalid argument: T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate d9caa1baaef84124a4a530b35945fad3 for term 49 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0825 15:14:26.070524 6683 raft_consensus.cc:1640] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9caa1baaef84124a4a530b35945fad3 in term 49. I0825 15:14:26.070688 6383 leader_election.cc:346] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Vote granted by peer 401cb2671e94452f87838515c7408bdc I0825 15:14:26.070724 6383 leader_election.cc:247] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. I0825 15:14:26.071020 6836 raft_consensus.cc:1931] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.961s I0825 15:14:26.071069 6836 raft_consensus.cc:1822] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Leader election won for term 49 I0825 15:14:26.071454 6836 raft_consensus.cc:443] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 LEADER]: Becoming Leader. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0825 15:14:26.071543 6836 consensus_queue.cc:145] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 49, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:26.072968 6433 catalog_manager.cc:1650] Tablet: f32f6e0b1f354643b6b030599a359fa6 reported consensus state change. New consensus state: current_term: 49 leader_uuid: "d9caa1baaef84124a4a530b35945fad3" config { opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } } I0825 15:14:26.208367 6799 raft_consensus.cc:363] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 48 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:14:26.208408 6799 raft_consensus.cc:1980] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 48 FOLLOWER]: Advancing to term 49 I0825 15:14:26.208806 6799 raft_consensus.cc:1931] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.148s I0825 15:14:26.208853 6799 raft_consensus.cc:379] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:26.209414 6799 leader_election.cc:222] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Requesting vote from peer 92f34d06b19146db82cf42701d2363c5 I0825 15:14:26.209504 6799 leader_election.cc:222] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Requesting vote from peer 401cb2671e94452f87838515c7408bdc I0825 15:14:26.209815 6683 raft_consensus.cc:1980] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 48 FOLLOWER]: Advancing to term 49 I0825 15:14:26.209853 6674 raft_consensus.cc:1980] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 48 FOLLOWER]: Advancing to term 49 I0825 15:14:26.210163 6683 raft_consensus.cc:1931] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.348s I0825 15:14:26.210187 6674 raft_consensus.cc:1931] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.880s I0825 15:14:26.210497 6674 raft_consensus.cc:1640] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9caa1baaef84124a4a530b35945fad3 in term 49. I0825 15:14:26.210515 6683 raft_consensus.cc:1640] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9caa1baaef84124a4a530b35945fad3 in term 49. I0825 15:14:26.210719 6383 leader_election.cc:346] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Vote granted by peer 401cb2671e94452f87838515c7408bdc I0825 15:14:26.210755 6383 leader_election.cc:247] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. I0825 15:14:26.211024 6383 leader_election.cc:346] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [CANDIDATE]: Term 49 election: Vote granted by peer 92f34d06b19146db82cf42701d2363c5 I0825 15:14:26.211025 6841 raft_consensus.cc:1931] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.090s I0825 15:14:26.211089 6841 raft_consensus.cc:1822] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Leader election won for term 49 I0825 15:14:26.211385 6841 raft_consensus.cc:443] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 LEADER]: Becoming Leader. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 1 Committed: term: 0 index: 0} I0825 15:14:26.211477 6841 consensus_queue.cc:145] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 49, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:26.212826 6433 catalog_manager.cc:1650] Tablet: 17aec689be0047b987494787029f6e08 reported consensus state change. New consensus state: current_term: 49 leader_uuid: "d9caa1baaef84124a4a530b35945fad3" config { opid_index: -1 local: false peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } } I0825 15:14:26.215440 6784 raft_consensus.cc:363] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0825 15:14:26.215481 6784 raft_consensus.cc:1980] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Advancing to term 50 I0825 15:14:26.215925 6784 raft_consensus.cc:1931] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.289s I0825 15:14:26.215967 6784 raft_consensus.cc:379] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:26.216498 6784 leader_election.cc:222] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 50 election: Requesting vote from peer d9caa1baaef84124a4a530b35945fad3 I0825 15:14:26.216593 6784 leader_election.cc:222] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 50 election: Requesting vote from peer 401cb2671e94452f87838515c7408bdc I0825 15:14:26.216953 6506 raft_consensus.cc:1603] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 LEADER]: Leader election vote request: Denying vote to candidate 92f34d06b19146db82cf42701d2363c5 for term 50 because replica is either leader or believes a valid leader to be alive. I0825 15:14:26.216970 6683 raft_consensus.cc:1980] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Advancing to term 50 I0825 15:14:26.217116 6436 leader_election.cc:360] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 50 election: Vote denied by peer d9caa1baaef84124a4a530b35945fad3. Message: Invalid argument: T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 LEADER]: Leader election vote request: Denying vote to candidate 92f34d06b19146db82cf42701d2363c5 for term 50 because replica is either leader or believes a valid leader to be alive. I0825 15:14:26.217299 6683 raft_consensus.cc:1931] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.096s I0825 15:14:26.217631 6683 raft_consensus.cc:1640] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f34d06b19146db82cf42701d2363c5 in term 50. I0825 15:14:26.217846 6445 leader_election.cc:346] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 50 election: Vote granted by peer 401cb2671e94452f87838515c7408bdc I0825 15:14:26.217875 6445 leader_election.cc:247] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. I0825 15:14:26.218158 6846 raft_consensus.cc:1931] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.608s I0825 15:14:26.218191 6846 raft_consensus.cc:1822] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 FOLLOWER]: Leader election won for term 50 I0825 15:14:26.218484 6846 raft_consensus.cc:443] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 LEADER]: Becoming Leader. State: Replica: 92f34d06b19146db82cf42701d2363c5, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 1 Committed: term: 0 index: 0} I0825 15:14:26.218571 6846 consensus_queue.cc:145] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 50, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } I0825 15:14:26.219779 6433 catalog_manager.cc:1650] Tablet: f32f6e0b1f354643b6b030599a359fa6 reported consensus state change. New consensus state: current_term: 50 leader_uuid: "92f34d06b19146db82cf42701d2363c5" config { opid_index: -1 local: false peers { permanent_uuid: "d9caa1baaef84124a4a530b35945fad3" member_type: VOTER last_known_addr { host: "127.108.26.0" port: 37291 } } peers { permanent_uuid: "92f34d06b19146db82cf42701d2363c5" member_type: VOTER last_known_addr { host: "127.108.26.1" port: 33958 } } peers { permanent_uuid: "401cb2671e94452f87838515c7408bdc" member_type: VOTER last_known_addr { host: "127.108.26.2" port: 35954 } } } I0825 15:14:26.377327 6683 raft_consensus.cc:800] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 57 FOLLOWER]: Rejecting Update request from peer d9caa1baaef84124a4a530b35945fad3 for earlier term 56. Current term is 57. Ops: [] I0825 15:14:26.377480 6674 raft_consensus.cc:800] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 LEADER]: Rejecting Update request from peer d9caa1baaef84124a4a530b35945fad3 for earlier term 56. Current term is 57. Ops: [] I0825 15:14:26.377887 6823 consensus_queue.cc:587] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Peer responded invalid term: Peer: 401cb2671e94452f87838515c7408bdc, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.378006 6851 consensus_queue.cc:587] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Peer responded invalid term: Peer: 92f34d06b19146db82cf42701d2363c5, Is new: false, Last received: 57.2, Next index: 3, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.378108 6827 raft_consensus.cc:1975] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 LEADER]: Stepping down as leader of term 56 I0825 15:14:26.378139 6827 raft_consensus.cc:478] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: LEADER Watermarks: {Received: term: 56 index: 1 Committed: term: 0 index: 0} I0825 15:14:26.378213 6827 consensus_queue.cc:162] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 56.1, Current term: 56, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:26.378418 6827 consensus_peers.cc:351] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 -> Peer 92f34d06b19146db82cf42701d2363c5 (127.108.26.1:33958): Closing peer: 92f34d06b19146db82cf42701d2363c5 I0825 15:14:26.378535 6827 consensus_peers.cc:351] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 -> Peer 401cb2671e94452f87838515c7408bdc (127.108.26.2:35954): Closing peer: 401cb2671e94452f87838515c7408bdc I0825 15:14:26.378628 6827 raft_consensus.cc:1980] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 56 FOLLOWER]: Advancing to term 57 I0825 15:14:26.427543 6683 raft_consensus.cc:832] T 1f7a6bd64c604f7fada909bd575708cd P 401cb2671e94452f87838515c7408bdc [term 57 FOLLOWER]: Refusing update from remote peer 92f34d06b19146db82cf42701d2363c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 57 index: 2. (index mismatch) I0825 15:14:26.427814 6506 raft_consensus.cc:832] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 57 FOLLOWER]: Refusing update from remote peer 92f34d06b19146db82cf42701d2363c5: Log matching property violated. Preceding OpId in replica: term: 56 index: 1. Preceding OpId from leader: term: 57 index: 2. (index mismatch) I0825 15:14:26.427917 6852 consensus_queue.cc:577] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Connected to new peer: Peer: 401cb2671e94452f87838515c7408bdc, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.428145 6852 log_cache.cc:306] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Successfully read 1 ops from disk (1..1) I0825 15:14:26.428247 6831 consensus_queue.cc:563] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Peer d9caa1baaef84124a4a530b35945fad3 log is divergent from this leader: its last log entry 56.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0 I0825 15:14:26.428273 6831 consensus_queue.cc:577] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Connected to new peer: Peer: d9caa1baaef84124a4a530b35945fad3, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.428395 6831 log_cache.cc:306] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5: Successfully read 1 ops from disk (1..1) I0825 15:14:26.428912 6506 raft_consensus_state.cc:401] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 57 FOLLOWER]: Aborting all transactions after (but not including): 0. Current State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: FOLLOWER Watermarks: {Received: term: 56 index: 1 Committed: term: 0 index: 0} I0825 15:14:26.428948 6506 raft_consensus_state.cc:428] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3 [term 57 FOLLOWER]: Aborting uncommitted NO_OP operation due to leader change: 56.1 I0825 15:14:26.428978 6506 raft_consensus.cc:1869] T 1f7a6bd64c604f7fada909bd575708cd P d9caa1baaef84124a4a530b35945fad3: NO_OP replication failed: Aborted: Transaction aborted by new leader I0825 15:14:26.430374 6854 raft_consensus_state.cc:534] T 1f7a6bd64c604f7fada909bd575708cd P 92f34d06b19146db82cf42701d2363c5 [term 57 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 57 index: 2 I0825 15:14:26.572681 6674 raft_consensus.cc:800] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 LEADER]: Rejecting Update request from peer d9caa1baaef84124a4a530b35945fad3 for earlier term 49. Current term is 50. Ops: [] I0825 15:14:26.572831 6683 raft_consensus.cc:800] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Rejecting Update request from peer d9caa1baaef84124a4a530b35945fad3 for earlier term 49. Current term is 50. Ops: [] I0825 15:14:26.572980 6855 consensus_queue.cc:587] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Peer responded invalid term: Peer: 92f34d06b19146db82cf42701d2363c5, Is new: false, Last received: 50.2, Next index: 3, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.573099 6840 raft_consensus.cc:1975] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 LEADER]: Stepping down as leader of term 49 I0825 15:14:26.573150 6840 raft_consensus.cc:478] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: LEADER Watermarks: {Received: term: 49 index: 1 Committed: term: 0 index: 0} I0825 15:14:26.573196 6836 consensus_queue.cc:587] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Peer responded invalid term: Peer: 401cb2671e94452f87838515c7408bdc, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.573277 6840 consensus_queue.cc:162] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 49.1, Current term: 49, Majority size: -1, State: 1, Mode: NON_LEADER I0825 15:14:26.573480 6840 consensus_peers.cc:351] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 -> Peer 401cb2671e94452f87838515c7408bdc (127.108.26.2:35954): Closing peer: 401cb2671e94452f87838515c7408bdc I0825 15:14:26.573621 6840 consensus_peers.cc:351] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 -> Peer 92f34d06b19146db82cf42701d2363c5 (127.108.26.1:33958): Closing peer: 92f34d06b19146db82cf42701d2363c5 I0825 15:14:26.573690 6840 raft_consensus.cc:1980] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 49 FOLLOWER]: Advancing to term 50 I0825 15:14:26.712508 6674 raft_consensus.cc:832] T 17aec689be0047b987494787029f6e08 P 92f34d06b19146db82cf42701d2363c5 [term 49 FOLLOWER]: Refusing update from remote peer d9caa1baaef84124a4a530b35945fad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 49 index: 2. (index mismatch) I0825 15:14:26.712667 6683 raft_consensus.cc:832] T 17aec689be0047b987494787029f6e08 P 401cb2671e94452f87838515c7408bdc [term 49 FOLLOWER]: Refusing update from remote peer d9caa1baaef84124a4a530b35945fad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 49 index: 2. (index mismatch) I0825 15:14:26.712795 6856 consensus_queue.cc:577] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Connected to new peer: Peer: 92f34d06b19146db82cf42701d2363c5, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.712911 6856 log_cache.cc:306] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Successfully read 1 ops from disk (1..1) I0825 15:14:26.712965 6841 consensus_queue.cc:577] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [LEADER]: Connected to new peer: Peer: 401cb2671e94452f87838515c7408bdc, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.713101 6841 log_cache.cc:306] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3: Successfully read 1 ops from disk (1..1) I0825 15:14:26.715018 6859 raft_consensus_state.cc:534] T 17aec689be0047b987494787029f6e08 P d9caa1baaef84124a4a530b35945fad3 [term 49 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 49 index: 2 I0825 15:14:26.719573 6506 raft_consensus.cc:832] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Refusing update from remote peer 92f34d06b19146db82cf42701d2363c5: Log matching property violated. Preceding OpId in replica: term: 49 index: 1. Preceding OpId from leader: term: 50 index: 2. (index mismatch) I0825 15:14:26.719705 6683 raft_consensus.cc:832] T f32f6e0b1f354643b6b030599a359fa6 P 401cb2671e94452f87838515c7408bdc [term 50 FOLLOWER]: Refusing update from remote peer 92f34d06b19146db82cf42701d2363c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 50 index: 2. (index mismatch) I0825 15:14:26.719902 6846 consensus_queue.cc:563] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Peer d9caa1baaef84124a4a530b35945fad3 log is divergent from this leader: its last log entry 49.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0 I0825 15:14:26.719929 6846 consensus_queue.cc:577] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Connected to new peer: Peer: d9caa1baaef84124a4a530b35945fad3, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.720016 6846 log_cache.cc:306] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Successfully read 1 ops from disk (1..1) I0825 15:14:26.720150 6860 consensus_queue.cc:577] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [LEADER]: Connected to new peer: Peer: 401cb2671e94452f87838515c7408bdc, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0825 15:14:26.720271 6860 log_cache.cc:306] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5: Successfully read 1 ops from disk (1..1) I0825 15:14:26.720446 6506 raft_consensus_state.cc:401] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Aborting all transactions after (but not including): 0. Current State: Replica: d9caa1baaef84124a4a530b35945fad3, State: 1, Role: FOLLOWER Watermarks: {Received: term: 49 index: 1 Committed: term: 0 index: 0} I0825 15:14:26.720468 6506 raft_consensus_state.cc:428] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3 [term 50 FOLLOWER]: Aborting uncommitted NO_OP operation due to leader change: 49.1 I0825 15:14:26.720487 6506 raft_consensus.cc:1869] T f32f6e0b1f354643b6b030599a359fa6 P d9caa1baaef84124a4a530b35945fad3: NO_OP replication failed: Aborted: Transaction aborted by new leader I0825 15:14:26.721595 6862 raft_consensus_state.cc:534] T f32f6e0b1f354643b6b030599a359fa6 P 92f34d06b19146db82cf42701d2363c5 [term 50 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 50 index: 2 I0825 15:16:24.454344 6374 delta_tracker.cc:446] Flushing 33 deltas from DMS 0... I0825 15:16:24.456564 6374 delta_tracker.cc:398] Flushed delta block: 482005392666535073 ts range: [6029980730434019328, 6029980741912727552] I0825 15:16:24.456668 6374 delta_tracker.cc:405] Reopened delta block for read: 482005392666535073 I0825 15:16:24.459153 6374 maintenance_manager.cc:357] Time spent running FlushDeltaMemStoresOp(00000000000000000000000000000000): real 0.005s user 0.003s sys 0.000s