[IMPALA-5557] Disable rpc_default_keepalive_time_ms

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 2.12.0
• Component/s: Distributed Exec
• Labels: scalability
• Epic Color: ghx-label-2
• Assignee: Michael Ho
• Reporter: Mostafa Mokhtar
• Created: 21/Jun/17 21:45
• Updated: 02/Jan/18 22:39
• Resolved: 02/Jan/18 22:39

Description

Queries were still hitting the KDC hard and eventually failed, this is the sequence of events:

1. Start running a query
2. Impala backends send thousands of TGS_REQ requests to the KDC
3. Query would appear to make some progress
4. Some connections succeed others fail with Timeout exceeded waiting to connect
5. Then I noticed in the logs that idle connections get kill after 65 seconds
   reactor.cc:281] Timing out connection server connection from 10.17.229.14:41804 - it has been idle for 65.0002s
6. Query takes about 2 minutes and fail
7. By then most connections are released since they were idle for > 65 seconds
8. New queries go through the same process again and eventually fail

In order to reliably run on a large cluster I believe we need to:

1. Change the lifetime of idle connections, possibly extend it to ticket lifetime?
2. Create new connections before existing ones expire in a staggered fashion to avoid KDC related failures?

This is the flag rpc_default_keepalive_time_ms

Issue Links

• duplicates: IMPALA-5901 KRPC: Client connection negotiation failed
• relates to: KUDU-279 RPC fails if it is sent exactly as the keepalive timeout expires
• relates to: KUDU-2237 Allows idle server connection detection to be disabled