[KUDU-2656] ErrorType/TabletServerDiskErrorTest.TestRandomOpSequence is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.8.0
• Fix Version/s: NA
• Component/s: test
• Assignee: Andrew Wong
• Reporter: Adar Dembo
• Created: 08/Jan/19 05:53
• Updated: 03/Jun/20 18:12
• Resolved: 03/Jun/20 18:12

Description

Here's the stack trace:

cfile_reader.cc:593] Check failed: io_context 
    @     0x7f3c7ec31c37 gsignal at ??:0
    @     0x7f3c7ec35028 abort at ??:0
    @     0x7f3c890b664e kudu::cfile::CFileReader::HandleCorruption() at ??:0
    @     0x7f3c890b4546 kudu::cfile::CFileReader::InitOnce() at ??:0
    @     0x7f3c890cbfbb kudu::cfile::CFileReader::Init()::$_0::operator()() at ??:0
    @     0x7f3c890cbe7c _ZZN4kudu14KuduOnceLambda4InitIZNS_5cfile11CFileReader4InitEPKNS_2fs9IOContextEE3$_0EENS_6StatusET_ENKUlvE_clEv at ??:0
    @     0x7f3c898e2a80 __GI___pthread_once at ??:0
    @     0x7f3c890cba4a _ZSt9call_onceIZN4kudu14KuduOnceLambda4InitIZNS0_5cfile11CFileReader4InitEPKNS0_2fs9IOContextEE3$_0EENS0_6StatusET_EUlvE_JEEvRSt9once_flagOSB_DpOT0_ at ??:0
    @     0x7f3c890b7a42 kudu::KuduOnceLambda::Init<>() at ??:0
    @     0x7f3c890b3ad8 kudu::cfile::CFileReader::Init() at ??:0
    @     0x7f3c8dbdec05 kudu::tablet::DeltaFileReader::InitOnce() at ??:0
    @     0x7f3c8dbe3bd8 kudu::tablet::DeltaFileReader::Init()::$_1::operator()() at ??:0
    @     0x7f3c8dbe3a3c _ZZN4kudu14KuduOnceLambda4InitIZNS_6tablet15DeltaFileReader4InitEPKNS_2fs9IOContextEE3$_1EENS_6StatusET_ENKUlvE_clEv at ??:0
    @     0x7f3c898e2a80 __GI___pthread_once at ??:0
    @     0x7f3c8dbe360a _ZSt9call_onceIZN4kudu14KuduOnceLambda4InitIZNS0_6tablet15DeltaFileReader4InitEPKNS0_2fs9IOContextEE3$_1EENS0_6StatusET_EUlvE_JEEvRSt9once_flagOSB_DpOT0_ at ??:0
    @     0x7f3c8dbde9e2 kudu::KuduOnceLambda::Init<>() at ??:0
    @     0x7f3c8dbde88d kudu::tablet::DeltaFileReader::Init() at ??:0
    @     0x7f3c8dc7e447 kudu::tablet::DeltaTracker::ValidateDeltaOrder() at ??:0
    @     0x7f3c8dc7fe46 kudu::tablet::DeltaTracker::AtomicUpdateStores() at ??:0
    @     0x7f3c8dc2a6e0 kudu::tablet::MajorDeltaCompaction::UpdateDeltaTracker() at ??:0
    @     0x7f3c8daf54d1 kudu::tablet::DiskRowSet::MajorCompactDeltaStoresWithColumnIds() at ??:0
    @     0x7f3c8daf4905 kudu::tablet::DiskRowSet::MajorCompactDeltaStores() at ??:0
    @     0x7f3c8d966409 kudu::tablet::Tablet::CompactWorstDeltas() at ??:0
    @     0x7f3c8da178eb kudu::tablet::MajorDeltaCompactionOp::Perform() at ??:0
    @     0x7f3c8372eef7 kudu::MaintenanceManager::LaunchOp() at ??:0
    @     0x7f3c8373fde3 boost::_bi::bind_t<>::operator()() at ??:0

From code inspection, it appears that ValidateDeltaOrder (only used in DEBUG builds) may Init() new delta files, and since it passes a null IOContext, we eventually trigger a CHECK in the CFileReader code.