[IMPALA-11748] Hostname flag is not respected by KRPC

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 4.1.1
• Component/s: Backend
• Labels: Kerberos Kubernetes

Environment

**Host:**
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"f59f5c2fda36e4036b49ec027e556a15456108f0", GitTreeState:"clean", BuildDate:"2022-01-19T17:26:47Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

**Container:**
impala@coordinator:/opt/impala$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"

The container image was built from source on Ubuntu 18.04 LTS with the following commands:
./buildall.sh -release -ninja -notests
ninja docker_images

• Epic Color: ghx-label-12
• Assignee: Unassigned
• Reporter: Tóth Andor
• Created: 25/Nov/22 13:51
• Updated: 19/Dec/22 02:07

Description

I am trying to set up a Kerberized Impala cluster in Kubernetes (K8S). In K8S deployments, hostnames of pods (containers) are unknown in advance (random suffix), therefore service principals (SPN) have to be used, which are fixed ahead. Impala has a `-hostname` flag that could be used for this, but it seems, that it's not respected by Kudu RPC (KRPC) and it's still using the original hostname returned by the system. Therefore it won't look for the right principal, and won't find a matching keytab entry, and finally it will fail to set up a KRPC connection.

See the stack trace for the error message:

I1125 11:22:12.005645  2949 exec-env.cc:483] Starting KRPC service
E1125 11:22:12.006127  2949 authentication.cc:239]  (stacktrace:
    @          0x116cd54  impala::SaslLogCallback()
    @     0x7f173c893a71  sasl_seterror
    @     0x7f1737499024  (unknown)
    @     0x7f173749b9ae  (unknown)
    @     0x7f173c89255d  sasl_server_step
    @     0x7f173c892b17  sasl_server_start
    @          0x1a73486  kudu::rpc::WrapSaslCall()
    @          0x1a78b78  kudu::rpc::ServerNegotiation::PreflightCheckGSSAPI()
    @          0x1a54cb7  kudu::rpc::Messenger::AddAcceptorPool()
    @          0x11a1fd0  impala::RpcMgr::StartServices()
    @          0x11e0ded  impala::ExecEnv::StartKrpcService()
    @          0x141e3d5  impala::ImpalaServer::Start()
    @          0x1402ada  ImpaladMain()
    @           0xbdbf7a  main
    @     0x7f17392fbc86  __libc_start_main
    @           0xc82279  _start
) SASL message (General): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No key table entry found matching impala/coordinator.impala.hadoop.svc.cluster.local@)
E1125 11:22:12.045207  2949 impalad-main.cc:90] Impalad services did not start correctly, exiting.  Error: Failed to add acceptor pool: Runtime error: GSSAPI/Kerberos not properly configured: No key table
 entry found matching impala/coordinator.impala.hadoop.svc.cluster.local@

In the case above, the hostname for the service pricipal (SPN) is "impala-coordinator.hadoop.svc.cluster.local", and the pod's FQDN is  "coordinator.impala.hadoop.svc.cluster.local".