Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1325

TS crashes trying to remote bootstrap a non-running tablet

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.1, 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      This happened on a TS while YCSB was running against the cluster:

      E0205 08:59:11.157722 66927 consensus_queue.cc:348] T b98bed072a534c54a08794d61dbb2736 P 6f83d42f66c34337a12f93f39c31c5db [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 35582 is ahead of the local log (next sequential op: 35582). Destination peer: Peer: fbd5dc753e264f819f1bc24fdc724982, Is new: false, Last received: 240.35582, Next index: 35583, Last known committed idx: 35580, Last exchange result: SUCCESS, Needs remote bootstrap: false
      *** Aborted at 1454703705 (unix time) try "date -d @1454703705" if you are using GNU date ***
      PC: @           0x978cd3 kudu::log::LogReader::GetSegmentsSnapshot()
      *** SIGSEGV (@0x48) received by PID 107515 (TID 0x7fe3e2420700) from PID 72; stack trace: ***
          @       0x316de0f710 (unknown)
          @           0x978cd3 kudu::log::LogReader::GetSegmentsSnapshot()
          @           0x78f42c kudu::tserver::RemoteBootstrapSession::Init()
          @           0x788b98 kudu::tserver::RemoteBootstrapServiceImpl::BeginRemoteBootstrapSession()
          @           0x7f549a kudu::tserver::RemoteBootstrapServiceIf::Handle()
          @           0x9d60a8 kudu::rpc::ServicePool::RunThread()
          @          0x179da1a kudu::Thread::SuperviseThread()
          @       0x316de079d1 (unknown)
          @       0x316dae88fd (unknown)
          @                0x0 (unknown)
      

      This is where the error happens:

      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/atomicops-internals-x86.h:87
      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/atomicops-internals-x86.h:144
      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/spinlock.h:74
      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/util/locks.h:45
      /opt/toolchain/boost-pic-1.55.0/include/boost/thread/lock_guard.hpp:38
      /usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/consensus/log_reader.cc:397
      

      Todd Lipcon thinks this is a NULL log_reader that's being accessed during a bad remote bootstrap. I'm saving the server logs and will do more digging.

        Attachments

          Activity

            People

            • Assignee:
              adar Adar Dembo
              Reporter:
              jdcryans Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: