[IMPALA-3575] Impala should retry backend connection request and apply a send timeout

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.2
• Fix Version/s: Impala 2.7.0
• Component/s: Distributed Exec
• Target Version: Impala 2.5.4
• Assignee: Juan Yu
• Reporter: Juan Yu
• Created: 19/May/16 21:01
• Updated: 10/Jan/24 06:34
• Resolved: 20/Jul/16 16:02

Description

Impala doesn't retry backend thrift connection request. If for any reason (another end too busy, high concurrent requests, crapy network), it cannot open a connection, it will return failure and the query will fail.

It would be nice to add a configurable retry number to let impala retry connecting to increase the connection success in above mentioned situation.

Also there is no timeout for sending and receiving data. Impala will wait forever if send or recv don't return. This is a reasonable choice for recv because we want to use back pressure to slow down downstream operator if upstream one cannot catch up. but for I don't think it makes sense for send. send could hung. A relative large send timeout could help Impala detect connection issues and fail query asap.

ClientCache already supports both retry and timeout parameter when creating a connection. it's very easy to add some configs for retry and timeout.

Issue Links

• blocks: IMPALA-3380 Add TCP timeouts to all RPCs that don't block
• is related to: IMPALA-12699 Coordinator should retry GetPartialCatalogObject request and apply a recv timeout