[IMPALA-4749] Hit DCHECK in sorter for spilling query with scratch limit

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: crash resource-management
• Target Version: Impala 2.9.0
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 10/Jan/17 01:05
• Updated: 11/Jan/17 00:34
• Resolved: 11/Jan/17 00:34

Description

I hit a DCHECK when running the following query on the test minicluster:

set max_block_mgr_memory=64m;
set scratch_limit=24m;
select o_orderdate, o_custkey, o_comment
from tpch.orders
order by o_orderdate;

F0109 16:56:25.251039 27202 exec-node.cc:204] Check failed: mem_tracker()->consumption() == 0 (8388608 vs. 0) Leaked memory.
Fragment 784ce2a2e1ebd300:31263e8000000002: Total=8.00 MB Peak=115.47 MB
  SORT_NODE (id=1): Total=8.00 MB Peak=72.13 MB
  HDFS_SCAN_NODE (id=0): Total=0 Peak=43.49 MB
  CodeGen: Total=4.36 KB Peak=450.00 KB
*** Check failure stack trace: ***
    @          0x282ae6d  google::LogMessage::Fail()
    @          0x282d796  google::LogMessage::SendToLog()
    @          0x282a98d  google::LogMessage::Flush()
    @          0x282e23e  google::LogMessageFatal::~LogMessageFatal()
    @          0x165e469  impala::ExecNode::Close()
    @          0x17a0ddf  impala::SortNode::Close()
    @          0x19f00a5  impala::PlanFragmentExecutor::Close()
    @          0x19e7b65  impala::FragmentInstanceState::Exec()
    @          0x19f31df  impala::QueryExecMgr::ExecFInstance()
    @          0x19f672a  boost::_mfi::mf1<>::operator()()
    @          0x19f65cd  boost::_bi::list2<>::operator()<>()
    @          0x19f6291  boost::_bi::bind_t<>::operator()()
    @          0x19f5d68  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @          0x134c6ae  boost::function0<>::operator()()
    @          0x15f5d23  impala::Thread::SuperviseThread()
    @          0x15fccfc  boost::_bi::list4<>::operator()<>()
    @          0x15fcc3f  boost::_bi::bind_t<>::operator()()
    @          0x15fcb9a  boost::detail::thread_data<>::run()
    @          0x1a5c45a  thread_proxy
    @     0x7f1ccc53170a  start_thread
    @     0x7f1ccc26782d  (unknown)
Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n 
Wrote minidump to /home/tarmstrong/Impala/incubator-impala/logs/cluster/minidumps/impalad/13b81910-07d0-3765-4f496d56-3ed53e84.dmp

This only affects debug builds - on a release build the memory should eventually be cleaned up.