Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6927

Crash when click Fragment Instances on web page

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 3.0, Impala 2.12.0
    • Fix Version/s: Impala 3.0, Impala 2.13.0
    • Component/s: None
    • Labels:
      None

      Description

      When clicking on "Fragment Instances" in web page for a running query, the impalad crashed. Looks like a DCHECK was hit. In the log:

      F0425 06:34:30.543802  8803 coordinator-backend-state.cc:616] Check failed: instance_stats.Size() == fragments_.size() (9 vs. 3)
      

      The stacktrace is

      #0  0x00007ff0b0ef4c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
      #1  0x00007ff0b0ef8028 in __GI_abort () at abort.c:89
      #2  0x0000000003fb3294 in google::DumpStackTraceAndExit() ()
      #3  0x0000000003fa9d0d in google::LogMessage::Fail() ()
      #4  0x0000000003fab5b2 in google::LogMessage::SendToLog() ()
      #5  0x0000000003fa96e7 in google::LogMessage::Flush() ()
      #6  0x0000000003faccae in google::LogMessageFatal::~LogMessageFatal() ()
      #7  0x0000000002de751f in impala::Coordinator::BackendState::InstanceStatsToJson (this=0xd62ab00, value=0x7feff0f00050, document=0x7feff0f002f0)
          at /mnt/volume1/impala-orc/incubator-impala/be/src/runtime/coordinator-backend-state.cc:616
      #8  0x0000000002dd7314 in impala::Coordinator::FInstanceStatsToJson (this=0xe1b4340, doc=0x7feff0f002f0) at /mnt/volume1/impala-orc/incubator-impala/be/src/runtime/coordinator.cc:973
      #9  0x0000000001b20e02 in impala::ImpalaHttpHandler::QueryFInstancesHandler (this=0x73dbf70, args=..., document=0x7feff0f002f0)
          at /mnt/volume1/impala-orc/incubator-impala/be/src/service/impala-http-handler.cc:730
      #10 0x0000000001b23533 in (anonymous namespace)::<lambda(const auto:1&, auto:2*)>::operator()<std::map<std::basic_string<char>, std::basic_string<char> >, rapidjson::GenericDocument<rapidjson::UTF8<> > >(const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > &, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > *) const (__closure=0xdc10238, args=..., doc=0x7feff0f002f0)
          at /mnt/volume1/impala-orc/incubator-impala/be/src/service/impala-http-handler.cc:62
      #11 0x0000000001b23568 in boost::detail::function::void_function_obj_invoker2<(anonymous namespace)::MakeCallback(T*, const F&) [with T = impala::ImpalaHttpHandler; F = void (impala::ImpalaHttpHandler::*)(const std::map<std::basic_string<char>, std::basic_string<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<> >*); impala::Webserver::UrlCallback = boost::function<void(const std::map<std::basic_string<char>, std::basic_string<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<> >*)>]::<lambda(const auto:1&, auto:2*)>, void, const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<const std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>::invoke(boost::detail::function::function_buffer &, const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > &, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > *) (
          function_obj_ptr=..., a0=..., a1=0x7feff0f002f0) at /mnt/volume1/impala-orc/incubator-impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
      #12 0x0000000001c3724a in boost::function2<void, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>::operator() (this=0xdc10230, a0=..., a1=0x7feff0f002f0)
          at /mnt/volume1/impala-orc/incubator-impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
      #13 0x0000000001c35599 in impala::Webserver::RenderUrlWithTemplate (this=0xebf5a40, arguments=..., url_handler=..., output=0x7feff0f008e0, content_type=0x7feff0f00a5c)
          at /mnt/volume1/impala-orc/incubator-impala/be/src/util/webserver.cc:439
      #14 0x0000000001c350a6 in impala::Webserver::BeginRequestCallback (this=0xebf5a40, connection=0xd5ab000, request_info=0xd5ab000) at /mnt/volume1/impala-orc/incubator-impala/be/src/util/webserver.cc:410
      #15 0x0000000001c34b47 in impala::Webserver::BeginRequestCallbackStatic (connection=0xd5ab000) at /mnt/volume1/impala-orc/incubator-impala/be/src/util/webserver.cc:369
      #16 0x0000000001c47d8a in handle_request (conn=0xd5ab000) at /mnt/volume1/impala-orc/incubator-impala/be/src/thirdparty/squeasel/squeasel.c:3874
      #17 0x0000000001c49fb6 in process_new_connection (conn=0xd5ab000) at /mnt/volume1/impala-orc/incubator-impala/be/src/thirdparty/squeasel/squeasel.c:4518
      #18 0x0000000001c4a68a in worker_thread (thread_func_param=0xc505b80) at /mnt/volume1/impala-orc/incubator-impala/be/src/thirdparty/squeasel/squeasel.c:4650
      #19 0x00007ff0b128f184 in start_thread (arg=0x7feff0f02700) at pthread_create.c:312
      #20 0x00007ff0b0fbbffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      

       

      Not sure how to reproduce it in the minicluster. I was playing around in Impala-2.12.0-rc1 on my cluster. It can be reproduced stably. My actions are:

      [localhost:21000] default> compute incremental stats myTable;
      

      Then in the webpage, the following subquery will be running:

      SELECT COUNT(*), realdate FROM myTable GROUP BY realdate
      

      Click its details, then click the "Fragment Instances" tab and the impalad crashed.

        Attachments

          Activity

            People

            • Assignee:
              lv Lars Volker
              Reporter:
              stiga-huang Quanlong Huang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: