[IMPALA-12114] SSL Thrift connections disconnect if idle more than ~150 seconds

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.3.0
• Fix Version/s: Impala 4.3.0
• Component/s: Backend
• Epic Color: ghx-label-12
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 04/May/23 00:55
• Updated: 02/Jun/23 16:57
• Resolved: 02/Jun/23 16:57

Description

A test cluster ran into issues with idle connections being disconnected when using SSL.

This reproduces on my development environment with these steps:

1. Start Impala with SSL enabled

bin/start-impala-cluster.py --impalad_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost --idle_client_poll_period_s=30 -v=2" --state_store_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost" --catalogd_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost" --cluster_size=1

1. Connect with impala-shell

impala-shell --ssl

1. Leave this idle for 150+ seconds
   In the Impalad logs will be a statement like this:

I0503 22:11:53.233147 206554 impala-server.cc:2488] Connection 20470cb275a1d256:3d68601942f3179f from client 172.27.100.70:42540 to server hiveserver2-frontend closed. The connection had 2 associated session(s).

1. Run a statement in impala-shell and will show that it needs to reconnect

default> show tables;
Caught exception TSocket read 0 bytes, type=<class 'thrift.transport.TTransport.TTransportException'> in PingImpalaHS2Service. 
Caught exception [Errno 32] Broken pipe, type=<class 'socket.error'> in CloseSession. 
Warning: close session RPC failed: [Errno 32] Broken pipe, <class 'socket.error'>
Connection lost, reconnecting...
... then it retries and succeeds

Tracing through the code, it appears that this peek() call returns false:

      try {
        bytes_pending = input_->getTransport()->peek();
        break;
      } catch (const TTransportException& ttx) {

bytes_pending is false, and this causes the connection to be closed.

This doesn't seem to impact Impala with older Thrift versions, so maybe something changed in Thrift 0.16.

Issue Links

• relates to: THRIFT-5705 C++ TSSLSocket accumulates receive retry counts across read() calls