[IMPALA-5106] KRPC DCHECK hit when closing DataStreamRecvr

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.9.0
• Component/s: Distributed Exec
• Epic Color: ghx-label-8
• Assignee: Henry Robinson
• Reporter: Henry Robinson
• Created: 21/Mar/17 03:07
• Updated: 24/Mar/17 22:00
• Resolved: 24/Mar/17 22:00

Description

Looks like a RowBatch wasn't destroyed before the DataStreamRecvr was closed.

Close() can arrive while a TransmitData() RPC is in flight, because of the asynchronous nature of the new RPC flow. And recent changes have made the locking in AddBatch() quite fine-grained. So the mem_tracker might be destroyed during AddBatch(), before the RowBatch is finished with. Looking at the code, it could also be torn down before the RowBatch is created, which would lead to an exception as well.

F0320 18:11:23.188339  4263 mem-tracker.cc:194] Check failed: consumption_->current_value() == 0 (24544 vs. 0) DataStreamRecvr
    @          0x1755bab  impala::MemTracker::~MemTracker()
    @          0x1740cbc  boost::checked_delete<>()
    @          0x173ef55  boost::scoped_ptr<>::~scoped_ptr()
    @          0x173f43f  boost::scoped_ptr<>::reset()
    @          0x17aed5f  impala::DataStreamRecvr::Close()
    @          0x1ab0969  impala::ExchangeNode::Close()
    @          0x1e95d65  impala::PlanFragmentExecutor::Close()
    @          0x1e8dc83  impala::FragmentInstanceState::Exec()
    @          0x1e98e65  impala::QueryExecMgr::ExecFInstance()
    @          0x1e9bca2  boost::_mfi::mf1<>::operator()()
    @          0x1e9bb2b  boost::_bi::list2<>::operator()<>()
    @          0x1e9b65f  boost::_bi::bind_t<>::operator()()