[IMPALA-6205] monotime.cc:200] Check failed: Initialized()

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: Not Applicable
• Component/s: Distributed Exec
• Epic Color: ghx-label-1
• Assignee: Michael Ho
• Reporter: Michael Brown
• Created: 16/Nov/17 19:58
• Updated: 29/Nov/17 03:22
• Resolved: 29/Nov/17 03:22

Description

During a secure stress rampup, Impala hit a dcheck:

F1116 11:27:37.387002  5507 monotime.cc:200] Check failed: Initialized()
F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()

(gdb) bt
#0  0x0000003797032625 in raise () from /lib64/libc.so.6
#1  0x0000003797033e05 in abort () from /lib64/libc.so.6
#2  0x0000000002fcaae4 in google::DumpStackTraceAndExit() ()
#3  0x0000000002fc155d in google::LogMessage::Fail() ()
#4  0x0000000002fc2e02 in google::LogMessage::SendToLog() ()
#5  0x0000000002fc0f37 in google::LogMessage::Flush() ()
#6  0x0000000002fc44fe in google::LogMessageFatal::~LogMessageFatal() ()
#7  0x0000000001ef8d6d in kudu::MonoTime::GetDeltaSince (this=0x7f5c18395288, rhs=...) at be/src/kudu/util/monotime.cc:200
#8  0x0000000001ef964b in kudu::operator- (t_end=..., t_beg=...) at be/src/kudu/util/monotime.cc:322
#9  0x0000000001fa39bb in kudu::rpc::InboundCallTiming::QueueTime (this=0x7f5c18395280) at be/src/kudu/rpc/inbound_call.h:66
#10 0x0000000001fa2c4f in kudu::rpc::RpczStore::LogTrace (this=0x14920780, call=0x7f5c183950e0) at be/src/kudu/rpc/rpcz_store.cc:228
#11 0x0000000001fa296f in kudu::rpc::RpczStore::AddCall (this=0x14920780, call=0x7f5c183950e0) at be/src/kudu/rpc/rpcz_store.cc:197
#12 0x0000000001f9c70b in kudu::rpc::InboundCall::Respond (this=0x7f5c183950e0, response=..., is_success=false) at be/src/kudu/rpc/inbound_call.cc:143
#13 0x0000000001f9c1a4 in kudu::rpc::InboundCall::RespondFailure (this=0x7f5c183950e0, error_code=kudu::rpc::ErrorStatusPB_RpcErrorCodePB_ERROR_SERVER_TOO_BUSY, status=...)
    at be/src/kudu/rpc/inbound_call.cc:110
#14 0x00000000020f5dba in kudu::rpc::ServicePool::RejectTooBusy (this=0x15818300, c=0x7f5c183950e0) at be/src/kudu/rpc/service_pool.cc:120
#15 0x00000000020f6268 in kudu::rpc::ServicePool::QueueInboundCall (this=0x15818300, call=...) at be/src/kudu/rpc/service_pool.cc:153
#16 0x0000000001fbb1a8 in kudu::rpc::Messenger::QueueInboundCall (this=0x14f54c60, call=...) at be/src/kudu/rpc/messenger.cc:424
#17 0x0000000001fae6ee in kudu::rpc::Connection::HandleIncomingCall (this=0x1af48200, transfer=...) at be/src/kudu/rpc/connection.cc:577
#18 0x0000000001fadf1a in kudu::rpc::Connection::ReadHandler (this=0x1af48200, watcher=..., revents=1) at be/src/kudu/rpc/connection.cc:540
#19 0x0000000001fb3d62 in ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection, &(kudu::rpc::Connection::ReadHandler)> (loop=0x14fe4900, w=0x1af482c0, revents=1)
    at toolchain/libev-4.20/include/ev++.h:479
#20 0x0000000002fe6023 in ev_invoke_pending ()
#21 0x0000000002fe9664 in ev_run ()
#22 0x0000000001f1055f in ev::loop_ref::run (this=0x148501c8, flags=0) at toolchain/libev-4.20/include/ev++.h:211
#23 0x0000000001fccfca in kudu::rpc::ReactorThread::RunThread (this=0x148501c0) at be/src/kudu/rpc/reactor.cc:375
#24 0x0000000001fd8535 in boost::_mfi::mf0<void, kudu::rpc::ReactorThread>::operator() (this=0x14c70990, p=0x148501c0)
    at toolchain/boost-1.57.0-p3/include/boost/bind/mem_fn_template.hpp:49
#25 0x0000000001fd7e18 in boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list0> (this=0x14c709a0, f=..., a=...)
    at toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:253
#26 0x0000000001fd7583 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator() (this=0x14c70990)
    at toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#27 0x0000000001fd6954 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >, void>::invoke (
    function_obj_ptr=...) at toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#28 0x000000000177a76e in boost::function0<void>::operator() (this=0x14c70988) at toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#29 0x0000000001f15ccf in kudu::Thread::SuperviseThread (arg=0x14c70960) at be/src/kudu/util/thread.cc:591
#30 0x0000003797407aa1 in start_thread () from /lib64/libpthread.so.0
#31 0x00000037970e893d in clone () from /lib64/libc.so.6
(gdb)