[IMPALA-2099] BufferedBlockMgr DCHECK client->num_tmp_reserved_buffers_ == 0 failure w/ RM enabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.2
• Labels: resource-management
• Target Version: Impala 2.3.0
• Assignee: Matthew Jacobs
• Reporter: Matthew Jacobs
• Created: 24/Jun/15 23:07
• Updated: 20/Dec/15 00:05
• Resolved: 27/Aug/15 00:24

Description

On a 6 node cluster with RM (Yarn+Llama) enabled, running tpcds q46 results in a DCHECK failure. The query completes fine in the same cluster with RM disabled. (profile attached).

F0624 14:10:18.447474  2245 buffered-block-mgr.cc:259] Check failed: client->num_tmp_reserved_buffers_ == 0 (3 vs. 0) 

The core dump shows the following stack (missing symbols so not much else):

…
#6  0x0000000001fd3ecd in google::LogMessageFatal::~LogMessageFatal() ()
#7  0x00000000011d72da in impala::BufferedBlockMgr::TryAcquireTmpReservation(impala::BufferedBlockMgr::Client*, int) ()
#8  0x0000000001519597 in impala::BufferedTupleStream::PinStream(bool, bool*) ()
#9  0x000000000160f65b in impala::Status impala::PartitionedHashJoinNode::Partition::BuildHashTableInternal<false>(impala::RuntimeState*, bool*) ()
#10 0x0000000001601c92 in impala::PartitionedHashJoinNode::Partition::BuildHashTable(impala::RuntimeState*, bool*, bool) ()
#11 0x0000000001608a68 in impala::PartitionedHashJoinNode::BuildHashTables(impala::RuntimeState*) ()
#12 0x000000000160402f in impala::PartitionedHashJoinNode::ProcessBuildInput(impala::RuntimeState*, int) ()
#13 0x0000000001602cc6 in impala::PartitionedHashJoinNode::ConstructBuildSide(impala::RuntimeState*) ()
#14 0x00000000016412f5 in impala::BlockingJoinNode::BuildSideThread(impala::RuntimeState*, impala::Promise<impala::Status>*) ()
#15 0x0000000001642ff2 in boost::_mfi::mf2<void, impala::BlockingJoinNode, impala::RuntimeState*, impala::Promise<impala::Status>*>::operator()(impala::BlockingJoinNode*, impala::RuntimeState*, impala::Promise<impala::Status>*) const ()
…

Issue Links

• relates to: IMPALA-2072 DCHECK(client->num_tmp_reserved_buffers_ == 0) on test_spilling during repeated-runs