[KUDU-688] consensus peers do IO on reactor thread

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Public beta
• Component/s: consensus
• Target Version/s: Public beta
• Code Review: http://gerrit.sjc.cloudera.com:8080/6710
• Assignee: Mike Percy
• Reporter: Todd Lipcon
• Created: 02/Apr/15 02:03
• Updated: 21/May/15 17:43
• Resolved: 21/May/15 17:43

Description

I used ThreadRestrictions to disallow IO on reactor threads, and hit the following CHECK:

F0402 00:01:11.673882 20014 thread_restrictions.cc:42] Check failed: LoadTLS()->io_allowed Function marked as IO-only was called from a thread that disallows IO! If this thread really should be allowed to make IO calls, adjust the call to kudu::ThreadRestrictions::SetIOAllowed() in this thread's startup.

• • • Check failure stack trace: ***
      @ 0x7fd3cb458f4d google::LogMessage::Fail()
      @ 0x7fd3cb45aded google::LogMessage::SendToLog()
      @ 0x7fd3cb458b3c google::LogMessage::Flush()
      @ 0x7fd3cb45b70e google::LogMessageFatal::~LogMessageFatal()
      @ 0x7fd3c79a6dff kudu::ThreadRestrictions::AssertIOAllowed()
      @ 0x7fd3c790dd27 kudu::(anonymous namespace)::PosixRandomAccessFile::Read()
      @ 0x7fd3c7912398 kudu::env_util::ReadFully()
      @ 0x7fd3c9d5abc5 kudu::log::ReadableLogSegment::ReadEntryHeader()
      @ 0x7fd3c9d59b57 kudu::log::ReadableLogSegment::ReadEntryHeaderAndBatch()
      @ 0x7fd3c9d99797 kudu::log::LogReader::ReadBatchUsingIndexEntry()
      @ 0x7fd3c9d9a11e kudu::log::LogReader::ReadReplicatesInRange()
      @ 0x7fd3c9ea0d39 kudu::consensus::LogCache::ReadOps()
      @ 0x7fd3c9e72281 kudu::consensus::PeerMessageQueue::GetOpsFromCacheOrFallback()
      @ 0x7fd3c9e72f79 kudu::consensus::PeerMessageQueue::RequestForPeer()
      @ 0x7fd3c9e6703f kudu::consensus::Peer::SendNextRequest()
      @ 0x7fd3c9e68445 kudu::consensus::Peer::ProcessResponse()

I added an exception for now, but we should probably defer the SendNextRequest() call to the peer request threadpool.