[IMPALA-3670] Block is not deleted by sorter in some cases when spilling is disabled

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 2.7.0, Impala 2.6.2
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.7.0, Impala 2.6.2
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 03/Jun/16 20:22
• Updated: 08/Sep/16 16:27
• Resolved: 07/Jun/16 05:06

Description

Start an impala cluster with no scratch:

./bin/start-impala-cluster.py --impalad_args=--scratch_dirs=""

Run this query:

set max_block_mgr_memory=64m;
select o_orderdate, o_custkey, o_comment
      from tpch.orders
      order by o_orderdate;

It fails with:

F0603 12:10:04.079107  9045 exec-node.cc:193] Check failed: mem_tracker()->consumption() == 0 (8388608 vs. 0) Leaked memory.
Fragment 984bd41202d209d7:a8fc5a9ca891d96: Consumption=8.00 MB
  SORT_NODE (id=1): Consumption=8.00 MB
  HDFS_SCAN_NODE (id=0): Consumption=0
*** Check failure stack trace: ***
    @          0x27d789d  google::LogMessage::Fail()
    @          0x27da1c6  google::LogMessage::SendToLog()
    @          0x27d73bd  google::LogMessage::Flush()
    @          0x27dac6e  google::LogMessageFatal::~LogMessageFatal()
    @          0x162e715  impala::ExecNode::Close()
    @          0x1771407  impala::SortNode::Close()
    @          0x195137a  impala::PlanFragmentExecutor::Close()
    @          0x151a7bc  impala::FragmentMgr::FragmentExecState::Exec()
    @          0x1511f85  impala::FragmentMgr::FragmentThread()
    @          0x1515b32  boost::_mfi::mf1<>::operator()()
    @          0x15158ef  boost::_bi::list2<>::operator()<>()
    @          0x1515219  boost::_bi::bind_t<>::operator()()
    @          0x1514b82  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @          0x1338664  boost::function0<>::operator()()
    @          0x15cfc29  impala::Thread::SuperviseThread()
    @          0x15d672a  boost::_bi::list4<>::operator()<>()
    @          0x15d666d  boost::_bi::bind_t<>::operator()()
    @          0x15d6630  boost::detail::thread_data<>::run()
    @          0x1a082aa  thread_proxy
    @     0x7f4cd26c86aa  start_thread
    @     0x7f4ccf8a2e9d  (unknown)

I did some digging and it looks like sorted_var_len_blocks in UnpinAllBlocks() are not always deleted in the failure case.

Status Sorter::Run::UnpinAllBlocks() {
...
  vector<BufferedBlockMgr::Block*> sorted_var_len_blocks; // These can be leaked!
  ...
          RETURN_IF_ERROR(TryAddBlock(UNPIN_PREV, &sorted_var_len_blocks, &added)); // Leak happens if this fails, because a pinned block is left in the list.

Issue Links

• is related to: IMPALA-1346 Sorter fails with out-of-memory in queries with multiple spilling operators