[IMPALA-3944] Crash in SimpleScheduler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 2.7.0
• Component/s: Distributed Exec
• Labels: crash downgraded scheduler
• Target Version: Impala 2.7.0
• Assignee: Lars Volker
• Reporter: Tim Armstrong
• Created: 03/Aug/16 00:57
• Updated: 15/Mar/18 19:57
• Resolved: 25/Aug/16 20:28

Description

I hit this DCHECK running locally using my scanner fuzzing patch. I had this running for a few hours.

./bin/start-impala-cluster.py && while impala-py.test tests/query_test/test_scanners_fuzz.py -k parquet --exploration_strategy=exhaustive ; do echo yes; done

Log file created at: 2016/08/02 15:36:07
Running on machine: tarmstrong-box
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0802 15:36:07.669970 27070 logging.cc:118] stderr will be logged to this file.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/tarmstrong/Impala/incubator-impala/fe/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/tarmstrong/Impala/incubator-impala/testdata/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
F0802 17:12:39.516374 27265 simple-scheduler.cc:1071] Check failed: backend_map().find(backend_ip) != backend_map().end() 
*** Check failure stack trace: ***
    @          0x2837d4d  google::LogMessage::Fail()
    @          0x283a676  google::LogMessage::SendToLog()
    @          0x283786d  google::LogMessage::Flush()
    @          0x283b11e  google::LogMessageFatal::~LogMessageFatal()
    @          0x144e9c0  impala::SimpleScheduler::AssignmentCtx::RecordScanRangeAssignment()
    @          0x1449500  impala::SimpleScheduler::ComputeScanRangeAssignment()
    @          0x1448c5a  impala::SimpleScheduler::ComputeScanRangeAssignment()
    @          0x144b2df  impala::SimpleScheduler::Schedule()
    @          0x1526bd9  impala::ImpalaServer::QueryExecState::ExecQueryOrDmlRequest()
    @          0x1523a87  impala::ImpalaServer::QueryExecState::Exec()
    @          0x14a8397  impala::ImpalaServer::ExecuteInternal()
    @          0x14a7b3a  impala::ImpalaServer::Execute()
    @          0x1517ec9  impala::ImpalaServer::query()
    @          0x1903458  beeswax::BeeswaxServiceProcessor::process_query()
    @          0x19031a6  beeswax::BeeswaxServiceProcessor::dispatchCall()
    @          0x18ecebd  impala::ImpalaServiceProcessor::dispatchCall()
    @          0x1182742  apache::thrift::TDispatchProcessor::process()
    @          0x27eab9b  apache::thrift::server::TThreadPoolServer::Task::run()
    @          0x27d2769  apache::thrift::concurrency::ThreadManager::Worker::run()
    @          0x134e48d  impala::ThriftThread::RunRunnable()
    @          0x134fbdd  boost::_mfi::mf2<>::operator()()
    @          0x134fa73  boost::_bi::list3<>::operator()<>()
    @          0x134f7bf  boost::_bi::bind_t<>::operator()()
    @          0x134f6d2  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @          0x136e19c  boost::function0<>::operator()()
    @          0x161955f  impala::Thread::SuperviseThread()
    @          0x161ff78  boost::_bi::list4<>::operator()<>()
    @          0x161febb  boost::_bi::bind_t<>::operator()()
    @          0x161fe16  boost::detail::thread_data<>::run()
    @          0x1a68b5a  thread_proxy
    @     0x7f5f64e586aa  start_thread
    @     0x7f5f62032e9d  (unknown)
Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n 
Wrote minidump to /home/tarmstrong/Impala/incubator-impala/logs/cluster/minidumps/impalad/5b2e5d7e-d4d0-cf79-5b7a3426-763e1dd7.dmp