[IMPALA-7566] TAcceptQueueServer connection setup should have timeout

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: Impala 2.11.0, Impala 3.0, Impala 2.12.0
• Fix Version/s: Impala 2.11.0
• Component/s: Clients, Distributed Exec
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Michael Ho
• Created: 13/Sep/18 06:42
• Updated: 05/Feb/19 11:05
• Resolved: 26/Sep/18 18:25

Description

Currently, there is no timeout when establishing a connection with an Impala client. For instance, if a client freezes whether (intentionally or unintentionally) in the middle of connection establishment (e.g. Saslhandshake), the single thread in connection_setup_pool of TAcceptQueueServer will be stuck waiting for the client and thus all other clients trying to connect to Beeswax or HS2 port of Impalad will be stuck forever. Impala should consider adding a timeout on the socket during connection establishment phase with a client so as limit the amount of time the thread can be stuck.

One can try using "openssl s_client" command to connect to Impalad with TLS and Kerberos enabled and leave it opened. The thread doing the connection setup will be stuck in the stack below:

Thread 551 (Thread 0x7fddde563700 (LWP 166354)):
#0  0x0000003ce2a0e82d in read () from /lib64/libpthread.so.0
#1  0x0000003ce56dea71 in ?? () from /usr/lib64/libcrypto.so.10
#2  0x0000003ce56dcdc9 in BIO_read () from /usr/lib64/libcrypto.so.10
#3  0x0000003ce9a2c1df in ssl3_read_n () from /usr/lib64/libssl.so.10
#4  0x0000003ce9a2c8dd in ssl3_read_bytes () from /usr/lib64/libssl.so.10
#5  0x0000003ce9a281a0 in ?? () from /usr/lib64/libssl.so.10
#6  0x000000000208ede2 in apache::thrift::transport::TSSLSocket::read(unsigned char*, unsigned int) ()
#7  0x000000000208b6f3 in unsigned int apache::thrift::transport::readAll<apache::thrift::transport::TSocket>(apache::thrift::transport::TSocket&, unsigned char*, unsigned int) ()
#8  0x0000000000cb2aa9 in apache::thrift::transport::TSaslTransport::receiveSaslMessage(apache::thrift::transport::NegotiationStatus*, unsigned int*) ()
#9  0x0000000000cb03e4 in apache::thrift::transport::TSaslServerTransport::handleSaslStartMessage() ()
#10 0x0000000000cb2c23 in apache::thrift::transport::TSaslTransport::doSaslNegotiation() ()
#11 0x0000000000cb10b8 in apache::thrift::transport::TSaslServerTransport::Factory::getTransport(boost::shared_ptr<apache::thrift::transport::TTransport>) ()
#12 0x0000000000b13e47 in apache::thrift::server::TAcceptQueueServer::SetupConnection(boost::shared_ptr<apache::thrift::transport::TTransport>) ()
#13 0x0000000000b14932 in boost::detail::function::void_function_obj_invoker2<apache::thrift::server::TAcceptQueueServer::serve()::{lambda(int, boost::shared_ptr<apache::thrift::transport::TTransport> const&)#1}, void, int, boost::shared_ptr<apache::thrift::transport::TTransport> const&>::invoke(boost::detail::function::function_buffer&, int, boost::shared_ptr<apache::thrift::transport::TTransport> const&) ()
#14 0x0000000000b177f9 in impala::ThreadPool<boost::shared_ptr<apache::thrift::transport::TTransport> >::WorkerThread(int) ()
#15 0x0000000000d602af in impala::Thread::SuperviseThread(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::ThreadDebugInfo const*, impala::Promise<long>*) ()
#16 0x0000000000d60aaa in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::ThreadDebugInfo const*, impala::Promise<long>*), boost::_bi::list5<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
#17 0x00000000012d756a in thread_proxy ()
#18 0x0000003ce2a07aa1 in start_thread () from /lib64/libpthread.so.0
#19 0x0000003ce26e893d in clone () from /lib64/libc.so.6

Issue Links

• is duplicated by: IMPALA-5394 Set socket timeouts while opening TSaslTransport
• is related to: IMPALA-7638 Lower default timeout for connection setup
• is related to: IMPALA-8159 TAcceptQueueServer: Caught TException: invalid sasl status
• relates to: IMPALA-7565 Extends TAcceptQueueServer connection_setup_pool to be multi-threaded