Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7566

TAcceptQueueServer connection setup should have timeout

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.11.0, Impala 3.0, Impala 2.12.0
    • Fix Version/s: Impala 2.11.0
    • Component/s: Clients, Distributed Exec
    • Labels:
      None

      Description

      Currently, there is no timeout when establishing a connection with an Impala client. For instance, if a client freezes whether (intentionally or unintentionally) in the middle of connection establishment (e.g. Saslhandshake), the single thread in connection_setup_pool of TAcceptQueueServer will be stuck waiting for the client and thus all other clients trying to connect to Beeswax or HS2 port of Impalad will be stuck forever. Impala should consider adding a timeout on the socket during connection establishment phase with a client so as limit the amount of time the thread can be stuck.

      One can try using "openssl s_client" command to connect to Impalad with TLS and Kerberos enabled and leave it opened. The thread doing the connection setup will be stuck in the stack below:

      Thread 551 (Thread 0x7fddde563700 (LWP 166354)):
      #0  0x0000003ce2a0e82d in read () from /lib64/libpthread.so.0
      #1  0x0000003ce56dea71 in ?? () from /usr/lib64/libcrypto.so.10
      #2  0x0000003ce56dcdc9 in BIO_read () from /usr/lib64/libcrypto.so.10
      #3  0x0000003ce9a2c1df in ssl3_read_n () from /usr/lib64/libssl.so.10
      #4  0x0000003ce9a2c8dd in ssl3_read_bytes () from /usr/lib64/libssl.so.10
      #5  0x0000003ce9a281a0 in ?? () from /usr/lib64/libssl.so.10
      #6  0x000000000208ede2 in apache::thrift::transport::TSSLSocket::read(unsigned char*, unsigned int) ()
      #7  0x000000000208b6f3 in unsigned int apache::thrift::transport::readAll<apache::thrift::transport::TSocket>(apache::thrift::transport::TSocket&, unsigned char*, unsigned int) ()
      #8  0x0000000000cb2aa9 in apache::thrift::transport::TSaslTransport::receiveSaslMessage(apache::thrift::transport::NegotiationStatus*, unsigned int*) ()
      #9  0x0000000000cb03e4 in apache::thrift::transport::TSaslServerTransport::handleSaslStartMessage() ()
      #10 0x0000000000cb2c23 in apache::thrift::transport::TSaslTransport::doSaslNegotiation() ()
      #11 0x0000000000cb10b8 in apache::thrift::transport::TSaslServerTransport::Factory::getTransport(boost::shared_ptr<apache::thrift::transport::TTransport>) ()
      #12 0x0000000000b13e47 in apache::thrift::server::TAcceptQueueServer::SetupConnection(boost::shared_ptr<apache::thrift::transport::TTransport>) ()
      #13 0x0000000000b14932 in boost::detail::function::void_function_obj_invoker2<apache::thrift::server::TAcceptQueueServer::serve()::{lambda(int, boost::shared_ptr<apache::thrift::transport::TTransport> const&)#1}, void, int, boost::shared_ptr<apache::thrift::transport::TTransport> const&>::invoke(boost::detail::function::function_buffer&, int, boost::shared_ptr<apache::thrift::transport::TTransport> const&) ()
      #14 0x0000000000b177f9 in impala::ThreadPool<boost::shared_ptr<apache::thrift::transport::TTransport> >::WorkerThread(int) ()
      #15 0x0000000000d602af in impala::Thread::SuperviseThread(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::ThreadDebugInfo const*, impala::Promise<long>*) ()
      #16 0x0000000000d60aaa in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::ThreadDebugInfo const*, impala::Promise<long>*), boost::_bi::list5<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
      #17 0x00000000012d756a in thread_proxy ()
      #18 0x0000003ce2a07aa1 in start_thread () from /lib64/libpthread.so.0
      #19 0x0000003ce26e893d in clone () from /lib64/libc.so.6
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kwho Michael Ho
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: