[IMPALA-6548] RPCs that fail with 'SSL_read: Connection timed out' should also be retried

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.11.0
• Component/s: Distributed Exec
• Epic Color: ghx-label-1
• Assignee: Vincent Tran
• Reporter: Vincent Tran
• Created: 20/Feb/18 23:13
• Updated: 23/Dec/20 01:40

Description

With IMPALA-5537, we made 'SSL_read: Resource temporarily unavailable' IsRecvTimeoutTException(). And thus retried indefinitely:

https://github.com/apache/impala/blob/ac86e9d931fa6e5d273e51c26cd7c1d1ad2cc6ce/be/src/runtime/data-stream-sender.cc#L231-L234

It seems to me that failed RPC via 'SSL_read: Connection timed out' should also be retried.

I0214 20:11:07.518100  7633 status.cc:55] RPC Error: Client for host1:22000 hits an unexpected exception: SSL_read: Connection timed out, type: N6apache6thrift9transport13TSSLExceptionE rpc send completed: true
    @           0x83c5ea  impala::Status::Status()
    @           0xdfb74e  impala::ClientConnection<>::DoRpc<>()
    @           0xdf794d  impala::DataStreamSender::Channel::DoTransmitDataRpc()
    @           0xdf8214  impala::DataStreamSender::Channel::TransmitDataHelper()
    @           0xdf85c1  impala::DataStreamSender::Channel::TransmitData()
    @           0xdfa5e2  impala::ThreadPool<>::WorkerThread()
    @           0xbef309  impala::Thread::SuperviseThread()
    @           0xbefcc4  boost::detail::thread_data<>::run()
    @           0xe5810a  (unknown)
    @       0x302a607aa1  (unknown)
    @       0x302a2e8bcd  (unknown)
I0214 20:11:07.518122  7633 client-cache.cc:170] Broken Connection, destroy client for host1:22000
I0214 20:11:07.518227  7633 thrift-util.cc:123] SSL_shutdown: Broken pipe
E0214 20:11:07.518746  7632 data-stream-sender.cc:278] channel send status: RPC Error: Client for host1:22000 hits an unexpected exception: SSL_read: Connection timed out, type: N6apache6thrift9transport13TSSLExceptionE rpc send completed: true
I0214 20:11:07.618536  7632 query-state.cc:324] Instance completed. instance_id=a94c7a78a8e1a5d9:a5719ddd00000005 #in-flight=5 status=RPC_GENERAL_ERROR: RPC Error: Client for host1:22000 hits an unexpected exception: SSL_read: Connection timed out, type: N6apache6thrift9transport13TSSLExceptionE rpc send completed: true
I0214 20:11:07.618548  7632 query-state.cc:334] Cancel: query_id=a94c7a78a8e1a5d9:a5719ddd00000000

Issue Links

• is related to: IMPALA-5537 Impala does not retry RPCs that fail in SSL_read()