Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6118

Assertion failure in mem-tracker when releasing runtime filter memory

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.11.0
    • Fix Version/s: Impala 2.11.0
    • Component/s: Backend
    • Labels:

      Description

      In a build on jenkins.impala.io, I hit an assertion in mem-tracker.h. The log file is https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/506/artifact/Impala/logs_static/logs/ee_tests/impalad.FATAL/*view*/ and https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/506/artifact/Impala/logs_static/logs/ee_tests/impalad.ip-172-31-13-161.ubuntu.log.ERROR.20171025-012412.38084/*view*/ . The ERROR log is attached to this ticket.

      The stack trace was:

      F1025 02:20:43.786911 82485 mem-tracker.h:231] Check failed: tracker->consumption_->current_value() >= 0 (-1052615 vs. 0)
      Runtime Filter (Coordinator): Total=-1.00 MB Peak=1.00 MB
      *** Check failure stack trace: ***
          @          0x2f1e11d  google::LogMessage::Fail()
          @          0x2f1f9c2  google::LogMessage::SendToLog()
          @          0x2f1daf7  google::LogMessage::Flush()
          @          0x2f210be  google::LogMessageFatal::~LogMessageFatal()
          @          0x17425fb  impala::MemTracker::Release()
          @          0x1fa7e8b  impala::Coordinator::UpdateFilter()
          @          0x186e3cf  impala::ImpalaServer::UpdateFilter()
          @          0x18d824f  impala::ImpalaInternalService::UpdateFilter()
          @          0x1dda35a  impala::ImpalaInternalServiceProcessor::process_UpdateFilter()
          @          0x1dd8308  impala::ImpalaInternalServiceProcessor::dispatchCall()
          @          0x15410ea  apache::thrift::TDispatchProcessor::process()
          @          0x171042b  apache::thrift::server::TAcceptQueueServer::Task::run()
          @          0x170c307  impala::ThriftThread::RunRunnable()
          @          0x170da13  boost::_mfi::mf2<>::operator()()
          @          0x170d8a9  boost::_bi::list3<>::operator()<>()
          @          0x170d5f5  boost::_bi::bind_t<>::operator()()
          @          0x170d508  boost::detail::function::void_function_obj_invoker0<>::invoke()
          @          0x171bdfc  boost::function0<>::operator()()
          @          0x19f3393  impala::Thread::SuperviseThread()
          @          0x19fbf26  boost::_bi::list4<>::operator()<>()
          @          0x19fbe69  boost::_bi::bind_t<>::operator()()
          @          0x19fbe2c  boost::detail::thread_data<>::run()
          @          0x20a7c9a  thread_proxy
          @     0x7fe6536186ba  start_thread
          @     0x7fe65334e3dd  clone
      r.java:81)
      

      Tim Armstrong suggested:

      Yeah it's probably another consequence of
      https://issues.apache.org/jira/browse/IMPALA-5789. Maybe your patch changed
      the timing enough to trigger it.

      I think the bug might be related to using directory.capacity() as the
      argument to Release(). Calling directory.clear() after releasing the memory
      in FitlerState::Disable() won't necessarily deallocate the memory so we
      could end up releasing it twice.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tianyiwang Tianyi Wang
                Reporter:
                philip Philip Zeyliger
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: