Details
Description
We have a KuduCluster with 14 TabletServers, but three TabletServers is down and can not start.
Some message in the TabletServer Log:
I0227 14:17:50.588430 31388 tablet.cc:902] T 54bfd393d27644b1bfee93244923bcd9 Flush: entering stage 1 (old memrowset already frozen for inserts)
I0227 14:17:50.588471 31388 compaction.cc:877] Selected 1 rowsets to compact:
I0227 14:17:50.588477 31388 compaction.cc:880] memrowset(current size on disk: ~0 bytes)
I0227 14:17:50.588486 31388 tablet.cc:904] T 54bfd393d27644b1bfee93244923bcd9 Memstore in-memory size: 1759 bytes
I0227 14:17:50.588517 31388 tablet.cc:1159] T 54bfd393d27644b1bfee93244923bcd9 Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6095540421588967424 or (T in
)}]
I0227 14:17:50.589731 31388 multi_column_writer.cc:85] Opened CFile writer for column party_id[string NOT NULL]
I0227 14:17:50.589802 31388 multi_column_writer.cc:85] Opened CFile writer for column cust_name[string NULLABLE]
I0227 14:17:50.589859 31388 multi_column_writer.cc:85] Opened CFile writer for column cust_name_spell[string NULLABLE]
...
F0227 14:17:50.591899 31388 diskrowset.cc:216] Check failed: last_encoded_key_.size() > 0 (0 vs. 0)
-
-
- Check failure stack trace: ***
@ 0x833afd google::LogMessage::Fail()
@ 0x8359fd google::LogMessage::SendToLog()
@ 0x833639 google::LogMessage::Flush()
@ 0x83649f google::LogMessageFatal::~LogMessageFatal()
@ 0x9544ea kudu::tablet::DiskRowSetWriter::FinishAndReleaseBlocks()
@ 0x954d71 kudu::tablet::RollingDiskRowSetWriter::FinishCurrentWriter()
@ 0x956066 kudu::tablet::RollingDiskRowSetWriter::Finish()
@ 0x90e3fd kudu::tablet::Tablet::DoMergeCompactionOrFlush()
@ 0x910655 kudu::tablet::Tablet::FlushInternal()
@ 0x910a67 kudu::tablet::Tablet::FlushUnlocked()
@ 0x9a277c kudu::tablet::FlushMRSOp::Perform()
@ 0x1a3213f kudu::MaintenanceManager::LaunchOp()
@ 0x1a7c93e kudu::ThreadPool::DispatchThread()
@ 0x1a7775a kudu::Thread::SuperviseThread()
@ 0x3b0d8079d1 (unknown)
@ 0x3b0d4e8b6d (unknown)
@ (nil) (unknown)
- Check failure stack trace: ***
-