[IMPALA-4069] Introduce startup option to create and cache backend connections on startup

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: Impala 2.5.0
• Component/s: Distributed Exec
• Labels: scalability
• Epic Link: Impala Scalability Improvement
• Target Version: Impala 2.9.0
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 02/Sep/16 21:10
• Updated: 21/Dec/20 18:56
• Resolved: 21/Dec/20 18:56

Description

Add impalad startup flag specifying the number of connections per backend to create and cache.
After startup impala-server.backends.client-cache.total-clients should reflect number of backends x cached connections per backend.

[e-mail] description of the problem

Internal Impala network connections between nodes for query execution are not multiplexed. This means as the number of queries increase the number of network connections increases between Impala executors. With higher #nodes, the combination of query bursts and number of executors can lead to lots of new connections attempts. For example, a query with 10+joins on a 100-node cluster could require 1000+ connections simultaneously on coordinator.  When the spike is too high or if there is not sufficient CPU available to handle the bursts, this causes connection failures. 
The total number of connections does not seem to be the issue, but there is currently a practical limit on the number of simultaneous new concurrent connection TCP request spikes at once. 

Impala caches backend connections and reuse them later. With cache, the simultaneous spikes of new connection request is only those above previous established maximum.

Issue Links

• is related to: IMPALA-4135 Thrift threaded server times-out waiting connections during high load
• relates to: IMPALA-2567 KRPC milestone 1