Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4069

Introduce startup option to create and cache backend connections on startup

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: None
    • Component/s: Distributed Exec
    • Labels:

      Description

      Add impalad startup flag specifying the number of connections per backend to create and cache.
      After startup impala-server.backends.client-cache.total-clients should reflect number of backends x cached connections per backend.

      Juan Yu description of the problem

      Internal Impala network connections between nodes for query execution are not multiplexed. This means as the number of queries increase the number of network connections increases between Impala executors. With higher #nodes, the combination of query bursts and number of executors can lead to lots of new connections attempts. For example, a query with 10+joins on a 100-node cluster could require 1000+ connections simultaneously on coordinator.  When the spike is too high or if there is not sufficient CPU available to handle the bursts, this causes connection failures. 
      The total number of connections does not seem to be the issue, but there is currently a practical limit on the number of simultaneous new concurrent connection TCP request spikes at once. 
      
      Impala caches backend connections and reuse them later. With cache, the simultaneous spikes of new connection request is only those above previous established maximum.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                1 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated: