Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6205

monotime.cc:200] Check failed: Initialized()

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: Not Applicable
    • Component/s: Distributed Exec
    • Labels:
      None
    • Epic Color:
      ghx-label-1

      Description

      During a secure stress rampup, Impala hit a dcheck:

      F1116 11:27:37.387002  5507 monotime.cc:200] Check failed: Initialized()
      F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
      F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
      F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
      F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
      F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
      F1116 11:27:37.398807  5492 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.401952  5537 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.403470  5531 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423159  5551 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.423761  5497 monotime.cc:200] Check failed: Initialized() F1116 11:27:37.447439  5549 monotime.cc:200] Check failed: Initialized()
      
      (gdb) bt
      #0  0x0000003797032625 in raise () from /lib64/libc.so.6
      #1  0x0000003797033e05 in abort () from /lib64/libc.so.6
      #2  0x0000000002fcaae4 in google::DumpStackTraceAndExit() ()
      #3  0x0000000002fc155d in google::LogMessage::Fail() ()
      #4  0x0000000002fc2e02 in google::LogMessage::SendToLog() ()
      #5  0x0000000002fc0f37 in google::LogMessage::Flush() ()
      #6  0x0000000002fc44fe in google::LogMessageFatal::~LogMessageFatal() ()
      #7  0x0000000001ef8d6d in kudu::MonoTime::GetDeltaSince (this=0x7f5c18395288, rhs=...) at be/src/kudu/util/monotime.cc:200
      #8  0x0000000001ef964b in kudu::operator- (t_end=..., t_beg=...) at be/src/kudu/util/monotime.cc:322
      #9  0x0000000001fa39bb in kudu::rpc::InboundCallTiming::QueueTime (this=0x7f5c18395280) at be/src/kudu/rpc/inbound_call.h:66
      #10 0x0000000001fa2c4f in kudu::rpc::RpczStore::LogTrace (this=0x14920780, call=0x7f5c183950e0) at be/src/kudu/rpc/rpcz_store.cc:228
      #11 0x0000000001fa296f in kudu::rpc::RpczStore::AddCall (this=0x14920780, call=0x7f5c183950e0) at be/src/kudu/rpc/rpcz_store.cc:197
      #12 0x0000000001f9c70b in kudu::rpc::InboundCall::Respond (this=0x7f5c183950e0, response=..., is_success=false) at be/src/kudu/rpc/inbound_call.cc:143
      #13 0x0000000001f9c1a4 in kudu::rpc::InboundCall::RespondFailure (this=0x7f5c183950e0, error_code=kudu::rpc::ErrorStatusPB_RpcErrorCodePB_ERROR_SERVER_TOO_BUSY, status=...)
          at be/src/kudu/rpc/inbound_call.cc:110
      #14 0x00000000020f5dba in kudu::rpc::ServicePool::RejectTooBusy (this=0x15818300, c=0x7f5c183950e0) at be/src/kudu/rpc/service_pool.cc:120
      #15 0x00000000020f6268 in kudu::rpc::ServicePool::QueueInboundCall (this=0x15818300, call=...) at be/src/kudu/rpc/service_pool.cc:153
      #16 0x0000000001fbb1a8 in kudu::rpc::Messenger::QueueInboundCall (this=0x14f54c60, call=...) at be/src/kudu/rpc/messenger.cc:424
      #17 0x0000000001fae6ee in kudu::rpc::Connection::HandleIncomingCall (this=0x1af48200, transfer=...) at be/src/kudu/rpc/connection.cc:577
      #18 0x0000000001fadf1a in kudu::rpc::Connection::ReadHandler (this=0x1af48200, watcher=..., revents=1) at be/src/kudu/rpc/connection.cc:540
      #19 0x0000000001fb3d62 in ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection, &(kudu::rpc::Connection::ReadHandler)> (loop=0x14fe4900, w=0x1af482c0, revents=1)
          at toolchain/libev-4.20/include/ev++.h:479
      #20 0x0000000002fe6023 in ev_invoke_pending ()
      #21 0x0000000002fe9664 in ev_run ()
      #22 0x0000000001f1055f in ev::loop_ref::run (this=0x148501c8, flags=0) at toolchain/libev-4.20/include/ev++.h:211
      #23 0x0000000001fccfca in kudu::rpc::ReactorThread::RunThread (this=0x148501c0) at be/src/kudu/rpc/reactor.cc:375
      #24 0x0000000001fd8535 in boost::_mfi::mf0<void, kudu::rpc::ReactorThread>::operator() (this=0x14c70990, p=0x148501c0)
          at toolchain/boost-1.57.0-p3/include/boost/bind/mem_fn_template.hpp:49
      #25 0x0000000001fd7e18 in boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list0> (this=0x14c709a0, f=..., a=...)
          at toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:253
      #26 0x0000000001fd7583 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator() (this=0x14c70990)
          at toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
      #27 0x0000000001fd6954 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >, void>::invoke (
          function_obj_ptr=...) at toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
      #28 0x000000000177a76e in boost::function0<void>::operator() (this=0x14c70988) at toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
      #29 0x0000000001f15ccf in kudu::Thread::SuperviseThread (arg=0x14c70960) at be/src/kudu/util/thread.cc:591
      #30 0x0000003797407aa1 in start_thread () from /lib64/libpthread.so.0
      #31 0x00000037970e893d in clone () from /lib64/libc.so.6
      (gdb)
      

        Attachments

          Activity

            People

            • Assignee:
              kwho Michael Ho
              Reporter:
              mikeb Michael Brown
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: