Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Distributed Exec
    • Labels:
      None
    • Epic Color:
      ghx-label-8

      Description

      Looks like a RowBatch wasn't destroyed before the DataStreamRecvr was closed.

      Close() can arrive while a TransmitData() RPC is in flight, because of the asynchronous nature of the new RPC flow. And recent changes have made the locking in AddBatch() quite fine-grained. So the mem_tracker might be destroyed during AddBatch(), before the RowBatch is finished with. Looking at the code, it could also be torn down before the RowBatch is created, which would lead to an exception as well.

      F0320 18:11:23.188339  4263 mem-tracker.cc:194] Check failed: consumption_->current_value() == 0 (24544 vs. 0) DataStreamRecvr
          @          0x1755bab  impala::MemTracker::~MemTracker()
          @          0x1740cbc  boost::checked_delete<>()
          @          0x173ef55  boost::scoped_ptr<>::~scoped_ptr()
          @          0x173f43f  boost::scoped_ptr<>::reset()
          @          0x17aed5f  impala::DataStreamRecvr::Close()
          @          0x1ab0969  impala::ExchangeNode::Close()
          @          0x1e95d65  impala::PlanFragmentExecutor::Close()
          @          0x1e8dc83  impala::FragmentInstanceState::Exec()
          @          0x1e98e65  impala::QueryExecMgr::ExecFInstance()
          @          0x1e9bca2  boost::_mfi::mf1<>::operator()()
          @          0x1e9bb2b  boost::_bi::list2<>::operator()<>()
          @          0x1e9b65f  boost::_bi::bind_t<>::operator()()
      

        Activity

        Hide
        henryr Henry Robinson added a comment -

        Fixed by my patch for IMPALA-4856. (It's a little tricky tracking bugs on a branch where we can go back in time and rewrite the offending commit).

        Show
        henryr Henry Robinson added a comment - Fixed by my patch for IMPALA-4856 . (It's a little tricky tracking bugs on a branch where we can go back in time and rewrite the offending commit).
        Hide
        henryr Henry Robinson added a comment -

        Previously this was handled by a big lock that was shared between Close() and AddBatch() so neither could run concurrently. We can fix this by adding back such a lock, just for the purpose of ensuring mutual exclusion between these two methods.

        Show
        henryr Henry Robinson added a comment - Previously this was handled by a big lock that was shared between Close() and AddBatch() so neither could run concurrently. We can fix this by adding back such a lock, just for the purpose of ensuring mutual exclusion between these two methods.

          People

          • Assignee:
            henryr Henry Robinson
            Reporter:
            henryr Henry Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development