[IMPALA-6970] DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.13.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend
• Labels: crash
• Target Version: Impala 2.13.0, Impala 3.1.0
• Epic Color: ghx-label-8
• Assignee: Tim Armstrong
• Reporter: Sailesh Mukil
• Created: 03/May/18 22:30
• Updated: 21/May/20 13:22
• Resolved: 08/May/18 16:12

Description

Similar to IMPALA-6587, but the DCHECK failed in a slightly different way. Cannot tell if the root cause is the same as that though without further investigation.

FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes <= unused_reservation() (8388608 vs. 6291456) 
*** Check failure stack trace: ***
    @          0x4277c1d  google::LogMessage::Fail()
    @          0x42794c2  google::LogMessage::SendToLog()
    @          0x42775f7  google::LogMessage::Flush()
    @          0x427abbe  google::LogMessageFatal::~LogMessageFatal()
    @          0x1ef1343  impala::ReservationTracker::AllocateFromLocked()
    @          0x1ef111d  impala::ReservationTracker::AllocateFrom()
    @          0x1ee8c57  impala::BufferPool::Client::PrepareToAllocateBuffer()
    @          0x1ee5543  impala::BufferPool::AllocateBuffer()
    @          0x2f50f68  impala::io::DiskIoMgr::AllocateBuffersForRange()
    @          0x1f74762  impala::HdfsScanNodeBase::StartNextScanRange()
    @          0x1f6b052  impala::HdfsScanNode::ScannerThread()
    @          0x1f6a4ea  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
    @          0x1f6c5cc  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
    @          0x1bd4748  boost::function0<>::operator()()
    @          0x1ebf349  impala::Thread::SuperviseThread()
    @          0x1ec74e5  boost::_bi::list5<>::operator()<>()
    @          0x1ec7409  boost::_bi::bind_t<>::operator()()
    @          0x1ec73cc  boost::detail::thread_data<>::run()
    @          0x31a1f0a  thread_proxy
    @       0x36d1607851  (unknown)
    @       0x36d12e894d  (unknown)

Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53

Issue Links

• relates to: IMPALA-6587 Crash in DiskMgr::AllocateBuffersForRange