Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1325

TS crashes trying to remote bootstrap a non-running tablet

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.6.0
    • 0.7.1, 0.8.0
    • None
    • None

    Description

      This happened on a TS while YCSB was running against the cluster:

      E0205 08:59:11.157722 66927 consensus_queue.cc:348] T b98bed072a534c54a08794d61dbb2736 P 6f83d42f66c34337a12f93f39c31c5db [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 35582 is ahead of the local log (next sequential op: 35582). Destination peer: Peer: fbd5dc753e264f819f1bc24fdc724982, Is new: false, Last received: 240.35582, Next index: 35583, Last known committed idx: 35580, Last exchange result: SUCCESS, Needs remote bootstrap: false
      *** Aborted at 1454703705 (unix time) try "date -d @1454703705" if you are using GNU date ***
      PC: @           0x978cd3 kudu::log::LogReader::GetSegmentsSnapshot()
      *** SIGSEGV (@0x48) received by PID 107515 (TID 0x7fe3e2420700) from PID 72; stack trace: ***
          @       0x316de0f710 (unknown)
          @           0x978cd3 kudu::log::LogReader::GetSegmentsSnapshot()
          @           0x78f42c kudu::tserver::RemoteBootstrapSession::Init()
          @           0x788b98 kudu::tserver::RemoteBootstrapServiceImpl::BeginRemoteBootstrapSession()
          @           0x7f549a kudu::tserver::RemoteBootstrapServiceIf::Handle()
          @           0x9d60a8 kudu::rpc::ServicePool::RunThread()
          @          0x179da1a kudu::Thread::SuperviseThread()
          @       0x316de079d1 (unknown)
          @       0x316dae88fd (unknown)
          @                0x0 (unknown)
      

      This is where the error happens:

      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/atomicops-internals-x86.h:87
      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/atomicops-internals-x86.h:144
      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/spinlock.h:74
      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/util/locks.h:45
      /opt/toolchain/boost-pic-1.55.0/include/boost/thread/lock_guard.hpp:38
      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/consensus/log_reader.cc:397
      

      Todd Lipcon thinks this is a NULL log_reader that's being accessed during a bad remote bootstrap. I'm saving the server logs and will do more digging.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            adar Adar Dembo
            jdcryans Jean-Daniel Cryans
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment