[KUDU-1325] TS crashes trying to remote bootstrap a non-running tablet

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.7.1, 0.8.0
• Target Version/s: 0.8.0
• Code Review: http://gerrit.cloudera.org:8080/2265
• Assignee: Adar Dembo
• Reporter: Jean-Daniel Cryans
• Created: 06/Feb/16 00:09
• Updated: 03/Mar/16 00:00
• Resolved: 26/Feb/16 06:51

Description

This happened on a TS while YCSB was running against the cluster:

E0205 08:59:11.157722 66927 consensus_queue.cc:348] T b98bed072a534c54a08794d61dbb2736 P 6f83d42f66c34337a12f93f39c31c5db [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 35582 is ahead of the local log (next sequential op: 35582). Destination peer: Peer: fbd5dc753e264f819f1bc24fdc724982, Is new: false, Last received: 240.35582, Next index: 35583, Last known committed idx: 35580, Last exchange result: SUCCESS, Needs remote bootstrap: false
*** Aborted at 1454703705 (unix time) try "date -d @1454703705" if you are using GNU date ***
PC: @           0x978cd3 kudu::log::LogReader::GetSegmentsSnapshot()
*** SIGSEGV (@0x48) received by PID 107515 (TID 0x7fe3e2420700) from PID 72; stack trace: ***
    @       0x316de0f710 (unknown)
    @           0x978cd3 kudu::log::LogReader::GetSegmentsSnapshot()
    @           0x78f42c kudu::tserver::RemoteBootstrapSession::Init()
    @           0x788b98 kudu::tserver::RemoteBootstrapServiceImpl::BeginRemoteBootstrapSession()
    @           0x7f549a kudu::tserver::RemoteBootstrapServiceIf::Handle()
    @           0x9d60a8 kudu::rpc::ServicePool::RunThread()
    @          0x179da1a kudu::Thread::SuperviseThread()
    @       0x316de079d1 (unknown)
    @       0x316dae88fd (unknown)
    @                0x0 (unknown)

This is where the error happens:

/usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/atomicops-internals-x86.h:87
/usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/atomicops-internals-x86.h:144
/usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/gutil/spinlock.h:74
/usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/util/locks.h:45
/opt/toolchain/boost-pic-1.55.0/include/boost/thread/lock_guard.hpp:38
/usr/src/debug/kudu-0.7.0-kudu0.7.0-SNAPSHOT/src/kudu/consensus/log_reader.cc:397

tlipcon thinks this is a NULL log_reader that's being accessed during a bad remote bootstrap. I'm saving the server logs and will do more digging.