Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-359

Replica tablets may be bootstrapping even though tablet is online

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • M4
    • None
    • tserver
    • None

    Description

      Apparently when going to replica tablets first (like we do in DistConsensusTest.TestInsertOnNonLeader) we might catch a replica bootstrapping even though the tablet is online.

      Trace:

      [ RUN ] DistConsensusTest.TestInsertOnNonLeader
      I0618 04:12:09.755156 28306 fsmanager.cc:97] Generated new instance metadata in path /data/2/tmp/kudutest-jenkins/DistConsensusTest.TestInsertOnNonLeader.1403089929570523-28306/master-root/instance:
      uuid: "e525e509dc1541ec9d859afa9bf6ada4"
      format_stamp: "Formatted at 2014-06-18 11:12:09 on a1213.halxg.cloudera.com"
      I0618 04:12:09.755251 28306 fsmanager.cc:57] Opened local filesystem: /data/2/tmp/kudutest-jenkins/DistConsensusTest.TestInsertOnNonLeader.1403089929570523-28306/master-root
      uuid: "e525e509dc1541ec9d859afa9bf6ada4"
      format_stamp: "Formatted at 2014-06-18 11:12:09 on a1213.halxg.cloudera.com"
      I0618 04:12:09.896139 28306 tablet_bootstrap.cc:231] Tablet id 11111111111111111111111111111111: Bootstrap starting.
      I0618 04:12:09.896914 28306 tablet_bootstrap.cc:312] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one.
      I0618 04:12:09.897660 28306 tablet_bootstrap.cc:231] Tablet id 11111111111111111111111111111111: No bootstrap required, opened a new log
      I0618 04:12:10.131182 28306 tablet_bootstrap.cc:231] Tablet id 00000000000000000000000000000000: Bootstrap starting.
      I0618 04:12:10.131990 28306 tablet_bootstrap.cc:312] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one.
      I0618 04:12:10.132818 28306 tablet_bootstrap.cc:231] Tablet id 00000000000000000000000000000000: No bootstrap required, opened a new log
      I0618 04:12:10.362515 28306 webserver.cc:105] Starting webserver on 0.0.0.0:0
      I0618 04:12:10.362545 28306 webserver.cc:114] Document root disabled
      I0618 04:12:10.362862 28306 webserver.cc:194] Webserver started. Bound to: 0.0.0.0:45586
      I0618 04:12:10.469576 28306 fsmanager.cc:97] Generated new instance metadata in path /data/2/tmp/kudutest-jenkins/DistConsensusTest.TestInsertOnNonLeader.1403089929570523-28306/ts-0-root/instance:
      uuid: "d7d0db8a822b43c29207e9db57e526e7"
      format_stamp: "Formatted at 2014-06-18 11:12:10 on a1213.halxg.cloudera.com"
      I0618 04:12:10.475198 28306 fsmanager.cc:57] Opened local filesystem: /data/2/tmp/kudutest-jenkins/DistConsensusTest.TestInsertOnNonLeader.1403089929570523-28306/ts-0-root
      uuid: "d7d0db8a822b43c29207e9db57e526e7"
      format_stamp: "Formatted at 2014-06-18 11:12:10 on a1213.halxg.cloudera.com"
      I0618 04:12:10.477821 28306 webserver.cc:105] Starting webserver on 0.0.0.0:0
      I0618 04:12:10.477839 28306 webserver.cc:114] Document root disabled
      I0618 04:12:10.478193 28306 webserver.cc:194] Webserver started. Bound to: 0.0.0.0:48924
      I0618 04:12:10.478389 28306 maintenance_manager.cc:97] MaintenanceManager: targetting memory size of 37.702118 GB
      I0618 04:12:10.480191 3566 heartbeater.cc:226] Registering TS with master...
      I0618 04:12:10.480243 3566 heartbeater.cc:232] Sending a full tablet report to master...
      I0618 04:12:10.480546 3532 ts_manager.cc:65] Registered entirely new tablet server: permanent_uuid: "d7d0db8a822b43c29207e9db57e526e7" instance_seqno: 1403089930475221
      I0618 04:12:10.627854 28306 fsmanager.cc:97] Generated new instance metadata in path /data/2/tmp/kudutest-jenkins/DistConsensusTest.TestInsertOnNonLeader.1403089929570523-28306/ts-1-root/instance:
      uuid: "1d4c81594f9d4456a0085efafaa75118"
      format_stamp: "Formatted at 2014-06-18 11:12:10 on a1213.halxg.cloudera.com"
      I0618 04:12:10.631893 28306 fsmanager.cc:57] Opened local filesystem: /data/2/tmp/kudutest-jenkins/DistConsensusTest.TestInsertOnNonLeader.1403089929570523-28306/ts-1-root
      uuid: "1d4c81594f9d4456a0085efafaa75118"
      format_stamp: "Formatted at 2014-06-18 11:12:10 on a1213.halxg.cloudera.com"
      I0618 04:12:10.634443 28306 webserver.cc:105] Starting webserver on 0.0.0.0:0
      I0618 04:12:10.634459 28306 webserver.cc:114] Document root disabled
      I0618 04:12:10.634696 28306 webserver.cc:194] Webserver started. Bound to: 0.0.0.0:40724
      I0618 04:12:10.634842 28306 maintenance_manager.cc:97] MaintenanceManager: targetting memory size of 37.702118 GB
      I0618 04:12:10.636760 3594 heartbeater.cc:226] Registering TS with master...
      I0618 04:12:10.636811 3594 heartbeater.cc:232] Sending a full tablet report to master...
      I0618 04:12:10.637151 3534 ts_manager.cc:65] Registered entirely new tablet server: permanent_uuid: "1d4c81594f9d4456a0085efafaa75118" instance_seqno: 1403089930631913
      I0618 04:12:10.761605 28306 fsmanager.cc:97] Generated new instance metadata in path /data/2/tmp/kudutest-jenkins/DistConsensusTest.TestInsertOnNonLeader.1403089929570523-28306/ts-2-root/instance:
      uuid: "29ebc490c48b49ae9446f78e9919bf69"
      format_stamp: "Formatted at 2014-06-18 11:12:10 on a1213.halxg.cloudera.com"
      I0618 04:12:10.765723 28306 fsmanager.cc:57] Opened local filesystem: /data/2/tmp/kudutest-jenkins/DistConsensusTest.TestInsertOnNonLeader.1403089929570523-28306/ts-2-root
      uuid: "29ebc490c48b49ae9446f78e9919bf69"
      format_stamp: "Formatted at 2014-06-18 11:12:10 on a1213.halxg.cloudera.com"
      I0618 04:12:10.768421 28306 webserver.cc:105] Starting webserver on 0.0.0.0:0
      I0618 04:12:10.768436 28306 webserver.cc:114] Document root disabled
      I0618 04:12:10.768661 28306 webserver.cc:194] Webserver started. Bound to: 0.0.0.0:48718
      I0618 04:12:10.768846 28306 maintenance_manager.cc:97] MaintenanceManager: targetting memory size of 37.702118 GB
      I0618 04:12:10.770561 3623 heartbeater.cc:226] Registering TS with master...
      I0618 04:12:10.770606 3623 heartbeater.cc:232] Sending a full tablet report to master...
      I0618 04:12:10.770952 3536 ts_manager.cc:65] Registered entirely new tablet server: permanent_uuid: "29ebc490c48b49ae9446f78e9919bf69" instance_seqno: 1403089930765743
      I0618 04:12:10.771919 28306 mini_cluster.cc:163] 3 TS(s) registered with Master after 0.00224542s
      I0618 04:12:10.772009 28306 mini_cluster.cc:163] 3 TS(s) registered with Master after 1.0952e-05s
      I0618 04:12:10.774210 3537 catalog_manager.cc:381] CreateTable from

      {real_user=jenkins, eff_user=}

      at 127.0.0.1:60493:
      name: "TestTable"
      schema {
      columns

      { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION }

      columns

      { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION }

      columns

      { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION }

      }
      I0618 04:12:10.779414 3554 tablet_service.cc:239] Processing CreateTablet for tablet 8cf8904cf700421db441e4206b0ec7e1 (table=TestTable [id=98af135a299d4ff8a76dc37c8cbaaca9]), range=["", ""]
      I0618 04:12:10.779844 3582 tablet_service.cc:239] Processing CreateTablet for tablet 8cf8904cf700421db441e4206b0ec7e1 (table=TestTable [id=98af135a299d4ff8a76dc37c8cbaaca9]), range=["", ""]
      I0618 04:12:10.780015 3611 tablet_service.cc:239] Processing CreateTablet for tablet 8cf8904cf700421db441e4206b0ec7e1 (table=TestTable [id=98af135a299d4ff8a76dc37c8cbaaca9]), range=["", ""]
      I0618 04:12:10.944509 3554 ts_tablet_manager.cc:381] Registered tablet 8cf8904cf700421db441e4206b0ec7e1
      I0618 04:12:10.944531 3554 ts_tablet_manager.cc:276] Bootstrapping tablet: 8cf8904cf700421db441e4206b0ec7e1
      I0618 04:12:10.944653 3554 tablet_bootstrap.cc:231] Tablet id 8cf8904cf700421db441e4206b0ec7e1: Bootstrap starting.
      I0618 04:12:10.945433 3554 tablet_bootstrap.cc:312] No previous blocks or log segments found for tablet: 8cf8904cf700421db441e4206b0ec7e1 creating new one.
      I0618 04:12:10.946190 3554 tablet_bootstrap.cc:231] Tablet id 8cf8904cf700421db441e4206b0ec7e1: No bootstrap required, opened a new log
      I0618 04:12:10.946300 3554 ts_tablet_manager.cc:281] Times for tablet 8cf8904cf700421db441e4206b0ec7e1 bootstrap: real 0.002s user 0.001s sys 0.000s
      I0618 04:12:10.946488 3554 raft_consensus.cc:54] Created Raft consensus for peer Replica: d7d0db8a822b43c29207e9db57e526e7, State: 0, Role: NON_PARTICIPANT
      Watermarks:

      {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0}

      Num. outstanding commits: 0 IsLocked: 0
      I0618 04:12:10.946614 3554 raft_consensus.cc:68] T 8cf8904cf700421db441e4206b0ec7e1 P d7d0db8a822b43c29207e9db57e526e7 [FOLLOWER]: Started. Quorum: seqno: -1 local: false peers { permanent_uuid: "29ebc490c48b49ae9446f78e9919bf69" role: FOLLOWER last_known_addr

      { host: "127.0.0.1" port: 32988 }

      } peers { permanent_uuid: "d7d0db8a822b43c29207e9db57e526e7" role: FOLLOWER last_known_addr

      { host: "127.0.0.1" port: 49071 }

      } peers { permanent_uuid: "1d4c81594f9d4456a0085efafaa75118" role: LEADER last_known_addr

      { host: "127.0.0.1" port: 43917 }

      }
      I0618 04:12:10.983069 3554 ts_tablet_manager.cc:301] Times for tablet 8cf8904cf700421db441e4206b0ec7e1 startup: real 0.037s user 0.001s sys 0.000s
      I0618 04:12:11.324728 3582 ts_tablet_manager.cc:381] Registered tablet 8cf8904cf700421db441e4206b0ec7e1
      I0618 04:12:11.324753 3582 ts_tablet_manager.cc:276] Bootstrapping tablet: 8cf8904cf700421db441e4206b0ec7e1
      I0618 04:12:11.324884 3582 tablet_bootstrap.cc:231] Tablet id 8cf8904cf700421db441e4206b0ec7e1: Bootstrap starting.
      I0618 04:12:11.325585 3582 tablet_bootstrap.cc:312] No previous blocks or log segments found for tablet: 8cf8904cf700421db441e4206b0ec7e1 creating new one.
      I0618 04:12:11.326340 3582 tablet_bootstrap.cc:231] Tablet id 8cf8904cf700421db441e4206b0ec7e1: No bootstrap required, opened a new log
      I0618 04:12:11.326448 3582 ts_tablet_manager.cc:281] Times for tablet 8cf8904cf700421db441e4206b0ec7e1 bootstrap: real 0.002s user 0.001s sys 0.000s
      I0618 04:12:11.326622 3582 raft_consensus.cc:54] Created Raft consensus for peer Replica: 1d4c81594f9d4456a0085efafaa75118, State: 0, Role: NON_PARTICIPANT
      Watermarks:

      {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0}

      Num. outstanding commits: 0 IsLocked: 0
      I0618 04:12:11.327173 3582 raft_consensus.cc:169] T 8cf8904cf700421db441e4206b0ec7e1 P 1d4c81594f9d4456a0085efafaa75118 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=0): replicating to peers...
      W0618 04:12:11.328783 3580 consensus_peers.cc:387] Couldn't send request to peer: 29ebc490c48b49ae9446f78e9919bf69 Status: Not found: Tablet not found. Retrying in the next heartbeat period. Already tried 1 times.
      I0618 04:12:11.329149 3582 raft_consensus.cc:181] T 8cf8904cf700421db441e4206b0ec7e1 P 1d4c81594f9d4456a0085efafaa75118 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=0): committing...
      W0618 04:12:11.329797 3580 consensus_peers.cc:387] Couldn't send request to peer: 29ebc490c48b49ae9446f78e9919bf69 Status: Not found: Tablet not found. Retrying in the next heartbeat period. Already tried 2 times.
      I0618 04:12:11.329800 3582 raft_consensus.cc:204] T 8cf8904cf700421db441e4206b0ec7e1 P 1d4c81594f9d4456a0085efafaa75118 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=0): a majority of peers have accepted the new configuration. Proceeding.
      I0618 04:12:11.329879 3582 raft_consensus.cc:68] T 8cf8904cf700421db441e4206b0ec7e1 P 1d4c81594f9d4456a0085efafaa75118 [LEADER]: Started. Quorum: seqno: 0 local: false peers { permanent_uuid: "29ebc490c48b49ae9446f78e9919bf69" role: FOLLOWER last_known_addr

      { host: "127.0.0.1" port: 32988 }

      } peers { permanent_uuid: "d7d0db8a822b43c29207e9db57e526e7" role: FOLLOWER last_known_addr

      { host: "127.0.0.1" port: 49071 }

      } peers { permanent_uuid: "1d4c81594f9d4456a0085efafaa75118" role: LEADER last_known_addr

      { host: "127.0.0.1" port: 43917 }

      }
      I0618 04:12:11.406980 3582 ts_tablet_manager.cc:301] Times for tablet 8cf8904cf700421db441e4206b0ec7e1 startup: real 0.080s user 0.001s sys 0.001s
      I0618 04:12:11.666858 3611 ts_tablet_manager.cc:381] Registered tablet 8cf8904cf700421db441e4206b0ec7e1
      I0618 04:12:11.666885 3611 ts_tablet_manager.cc:276] Bootstrapping tablet: 8cf8904cf700421db441e4206b0ec7e1
      I0618 04:12:11.667062 3611 tablet_bootstrap.cc:231] Tablet id 8cf8904cf700421db441e4206b0ec7e1: Bootstrap starting.
      W0618 04:12:11.827553 3580 consensus_peers.cc:387] Couldn't send request to peer: 29ebc490c48b49ae9446f78e9919bf69 Status: Service unavailable: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times.
      W0618 04:12:11.848723 3631 reactor.cc:196] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:51844, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}}
      W0618 04:12:11.850612 3817 reactor.cc:196] Client call Write has no timeout set for connection id: {remote=127.0.0.1:49071, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}}
      W0618 04:12:11.852740 3817 reactor.cc:196] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43917, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}}
      W0618 04:12:11.854817 3817 reactor.cc:196] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:49071, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}}
      W0618 04:12:11.856016 3818 reactor.cc:196] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:32988, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}}
      /var/lib/jenkins/workspace/kudu-gerrit/BUILD_TYPE/DEBUG/label/centos6-kudu/src/integration-tests/raft_consensus-itest.cc:189: Failure
      Value of: resp.has_error()
      Actual: true
      Expected: false
      Google Test trace:
      /var/lib/jenkins/workspace/kudu-gerrit/BUILD_TYPE/DEBUG/label/centos6-kudu/src/integration-tests/raft_consensus-itest.cc:188: error {
      code: TABLET_NOT_RUNNING
      status

      { code: SERVICE_UNAVAILABLE message: "Tablet not RUNNING: BOOTSTRAPPING" }

      }

      /var/lib/jenkins/workspace/kudu-gerrit/BUILD_TYPE/DEBUG/label/centos6-kudu/src/integration-tests/raft_consensus-itest.cc:186: new_scan_request {
      tablet_id: "8cf8904cf700421db441e4206b0ec7e1"
      projected_columns

      { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION }

      projected_columns

      { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION }

      projected_columns

      { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION }

      }
      batch_size_bytes: 0

      /var/lib/jenkins/workspace/kudu-gerrit/BUILD_TYPE/DEBUG/label/centos6-kudu/src/integration-tests/raft_consensus-itest.cc:415: error {
      code: UNKNOWN_ERROR
      status

      { code: ILLEGAL_STATE message: "Replica is not leader of this quorum." }

      }

      I0618 04:12:11.857630 28306 tablet_server.cc:100] TabletServer shutting down...
      I0618 04:12:11.868839 28306 ts_tablet_manager.cc:336] Shutting down tablet manager...
      I0618 04:12:11.868887 28306 maintenance_manager.cc:160] Unregistered op FlushRowSetsOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:11.868904 28306 maintenance_manager.cc:160] Unregistered op CompactRowSetsOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:11.868921 28306 maintenance_manager.cc:160] Unregistered op FlushDeltaMemStoresOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:11.869101 28306 raft_consensus_state.cc:190] Replica d7d0db8a822b43c29207e9db57e526e7 waiting on 0 outstanding commits.
      I0618 04:12:11.869118 28306 raft_consensus_state.cc:194] All local commits completed for replica: d7d0db8a822b43c29207e9db57e526e7
      I0618 04:12:11.870666 28306 raft_consensus.cc:514] Raft consensus Shutdown!
      I0618 04:12:11.870789 28306 raft_consensus_state.cc:190] Replica d7d0db8a822b43c29207e9db57e526e7 waiting on 0 outstanding commits.
      I0618 04:12:11.870801 28306 raft_consensus_state.cc:194] All local commits completed for replica: d7d0db8a822b43c29207e9db57e526e7
      I0618 04:12:11.870823 28306 raft_consensus.cc:514] Raft consensus Shutdown!
      I0618 04:12:11.871572 28306 tablet_server.cc:109] TabletServer shut down complete. Bye!
      I0618 04:12:11.871587 28306 tablet_server.cc:100] TabletServer shutting down...
      W0618 04:12:11.871621 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      I0618 04:12:11.871635 28306 ts_tablet_manager.cc:336] Shutting down tablet manager...
      I0618 04:12:11.871659 28306 tablet_server.cc:109] TabletServer shut down complete. Bye!
      W0618 04:12:11.871819 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      W0618 04:12:11.871889 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      I0618 04:12:11.872107 28306 tablet_server.cc:100] TabletServer shutting down...
      I0618 04:12:11.883339 28306 ts_tablet_manager.cc:336] Shutting down tablet manager...
      I0618 04:12:11.883386 28306 maintenance_manager.cc:160] Unregistered op FlushRowSetsOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:11.883404 28306 maintenance_manager.cc:160] Unregistered op CompactRowSetsOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:11.883415 28306 maintenance_manager.cc:160] Unregistered op FlushDeltaMemStoresOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:11.883573 28306 consensus_peers.cc:407] Closing peer: 1d4c81594f9d4456a0085efafaa75118 Waiting for outstanding requests to complete 0
      I0618 04:12:11.883668 28306 consensus_peers.cc:407] Closing peer: d7d0db8a822b43c29207e9db57e526e7 Waiting for outstanding requests to complete 0
      I0618 04:12:11.883811 28306 consensus_peers.cc:407] Closing peer: 29ebc490c48b49ae9446f78e9919bf69 Waiting for outstanding requests to complete 0
      I0618 04:12:11.883889 28306 raft_consensus_state.cc:190] Replica 1d4c81594f9d4456a0085efafaa75118 waiting on 0 outstanding commits.
      I0618 04:12:11.883904 28306 raft_consensus_state.cc:194] All local commits completed for replica: 1d4c81594f9d4456a0085efafaa75118
      I0618 04:12:11.908807 3611 tablet_bootstrap.cc:312] No previous blocks or log segments found for tablet: 8cf8904cf700421db441e4206b0ec7e1 creating new one.
      I0618 04:12:11.910713 28306 raft_consensus.cc:514] Raft consensus Shutdown!
      I0618 04:12:11.910804 28306 raft_consensus_state.cc:190] Replica 1d4c81594f9d4456a0085efafaa75118 waiting on 0 outstanding commits.
      I0618 04:12:11.910823 28306 raft_consensus_state.cc:194] All local commits completed for replica: 1d4c81594f9d4456a0085efafaa75118
      I0618 04:12:11.910848 28306 raft_consensus.cc:514] Raft consensus Shutdown!
      I0618 04:12:11.911866 28306 tablet_server.cc:109] TabletServer shut down complete. Bye!
      I0618 04:12:11.911886 28306 tablet_server.cc:100] TabletServer shutting down...
      W0618 04:12:11.911937 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      I0618 04:12:11.911957 28306 ts_tablet_manager.cc:336] Shutting down tablet manager...
      I0618 04:12:11.911988 28306 tablet_server.cc:109] TabletServer shut down complete. Bye!
      W0618 04:12:11.912145 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      W0618 04:12:11.912247 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      I0618 04:12:11.912657 28306 tablet_server.cc:100] TabletServer shutting down...
      I0618 04:12:12.448390 3611 tablet_bootstrap.cc:231] Tablet id 8cf8904cf700421db441e4206b0ec7e1: No bootstrap required, opened a new log
      I0618 04:12:12.448547 3611 ts_tablet_manager.cc:281] Times for tablet 8cf8904cf700421db441e4206b0ec7e1 bootstrap: real 0.782s user 0.001s sys 0.001s
      I0618 04:12:12.448755 3611 raft_consensus.cc:54] Created Raft consensus for peer Replica: 29ebc490c48b49ae9446f78e9919bf69, State: 0, Role: NON_PARTICIPANT
      Watermarks:

      {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0}

      Num. outstanding commits: 0 IsLocked: 0
      I0618 04:12:12.448876 3611 raft_consensus.cc:68] T 8cf8904cf700421db441e4206b0ec7e1 P 29ebc490c48b49ae9446f78e9919bf69 [FOLLOWER]: Started. Quorum: seqno: -1 local: false peers { permanent_uuid: "29ebc490c48b49ae9446f78e9919bf69" role: FOLLOWER last_known_addr

      { host: "127.0.0.1" port: 32988 }

      } peers { permanent_uuid: "d7d0db8a822b43c29207e9db57e526e7" role: FOLLOWER last_known_addr

      { host: "127.0.0.1" port: 49071 }

      } peers { permanent_uuid: "1d4c81594f9d4456a0085efafaa75118" role: LEADER last_known_addr

      { host: "127.0.0.1" port: 43917 }

      }
      I0618 04:12:12.651188 3611 ts_tablet_manager.cc:301] Times for tablet 8cf8904cf700421db441e4206b0ec7e1 startup: real 0.203s user 0.000s sys 0.001s
      I0618 04:12:12.651628 28306 ts_tablet_manager.cc:336] Shutting down tablet manager...
      I0618 04:12:12.651691 28306 maintenance_manager.cc:160] Unregistered op FlushRowSetsOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:12.651710 28306 maintenance_manager.cc:160] Unregistered op CompactRowSetsOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:12.651721 28306 maintenance_manager.cc:160] Unregistered op FlushDeltaMemStoresOp(8cf8904cf700421db441e4206b0ec7e1)
      I0618 04:12:12.651921 28306 raft_consensus_state.cc:190] Replica 29ebc490c48b49ae9446f78e9919bf69 waiting on 0 outstanding commits.
      I0618 04:12:12.651942 28306 raft_consensus_state.cc:194] All local commits completed for replica: 29ebc490c48b49ae9446f78e9919bf69
      I0618 04:12:12.653516 28306 raft_consensus.cc:514] Raft consensus Shutdown!
      I0618 04:12:12.653575 28306 raft_consensus_state.cc:190] Replica 29ebc490c48b49ae9446f78e9919bf69 waiting on 0 outstanding commits.
      I0618 04:12:12.653587 28306 raft_consensus_state.cc:194] All local commits completed for replica: 29ebc490c48b49ae9446f78e9919bf69
      I0618 04:12:12.653609 28306 raft_consensus.cc:514] Raft consensus Shutdown!
      I0618 04:12:12.654218 28306 tablet_server.cc:109] TabletServer shut down complete. Bye!
      I0618 04:12:12.654243 28306 tablet_server.cc:100] TabletServer shutting down...
      W0618 04:12:12.654285 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      I0618 04:12:12.654299 28306 ts_tablet_manager.cc:336] Shutting down tablet manager...
      I0618 04:12:12.654320 28306 tablet_server.cc:109] TabletServer shut down complete. Bye!
      W0618 04:12:12.654551 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      W0618 04:12:12.654613 28306 rpc_server.cc:102] Unable to unregister service kudu.tserver.TabletServerService: Service unavailable: Service is not registered
      I0618 04:12:12.654923 28306 master.cc:71] Master@127.0.0.1:51844 shutting down...
      I0618 04:12:12.817142 28306 master.cc:74] Master@127.0.0.1:51844 shutdown complete.
      W0618 04:12:12.818436 28306 rpc_server.cc:102] Unable to unregister service kudu.master.MasterService: Service unavailable: Service is not registered
      W0618 04:12:12.818541 28306 rpc_server.cc:102] Unable to unregister service kudu.master.MasterService: Service unavailable: Service is not registered
      [ FAILED ] DistConsensusTest.TestInsertOnNonLeader (3248 ms)

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. KUDU-370 Raft consensus integration test TestInsertOnNonLeader fails if the target replica is CONFIGURING

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              dralves David Alves
              dralves David Alves
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: