[IMPALA-3189] Address scalability issue with N^2 KDC requests on cluster startup

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.5.0
• Component/s: Distributed Exec, Security
• Labels: kerberos scalability
• Epic Link: Impala Scalability Improvement
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Henry Robinson
• Created: 16/Mar/16 18:55
• Updated: 23/Dec/20 21:35

Description

When Impala runs a query that shuffles data amongst all nodes in a Kerberos-secured cluster, every node will need to acquire a TGS for every other node. In a cluster of 100 nodes or more, this can overwhelm the KDC, and queries can exit with an error ("Could not contact KDC for realm").

A simple workaround is to run a warm-up query until it succeeds (which can take a few minutes after cluster startup). The KDC can also be scaled (e.g. with secondary KDC nodes).

Impala can also consider either forcing a TGS request on start-up in a staggered fashion, or we can move to recommending SSL + client certificates for server<->server communication.

Issue Links

• duplicates: IMPALA-5020 Query against large cluster with SSL + Kerberos enabled failed with RPC client failed to connect: Couldn't open transport for foo:22000 (Could not resolve host for client socket.)
• duplicates: IMPALA-4860 Stagger impala TGS_REQ to KDC
• is duplicated by: IMPALA-5020 Query against large cluster with SSL + Kerberos enabled failed with RPC client failed to connect: Couldn't open transport for foo:22000 (Could not resolve host for client socket.)
• is related to: IMPALA-6720 impala deamon server 22000 connect refuse in cluster running time