[HIVE-21899] Utils.getCanonicalHostName() may return IP address depending on DNS infra

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0, 2.4.0, 3.0.0, 3.1.1
• Component/s: HiveServer2, Metastore, Security
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: KWON BYUNGCHANG
• Created: 20/Jun/19 04:00
• Updated: 26/Aug/22 07:33

Description

if there is not PTR record of hostname A in DNS,
org.apache.hive.jdbc.Utils.getCanonicalHostName(“A”) return IP Address.
And failed connecting secured HS2 or HMS because cannot getting kerberos service ticket of HS2 or HMS using ip address.

workaround is adding hostname A and IP to /etc/hosts, it is uncomfortable.

below is krb5 debug log.
note that Server not found in Kerberos database and hive/[e-mail]

Picked up JAVA_TOOL_OPTIONS: -Dsun.security.krb5.debug=true

Connecting to jdbc:hive2://zk1.example.com:2181,zk2.example.com:2181,zk.example.com:2181/default;principal=hive/[e-mail];serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Java config name: /etc/krb5.conf
Loaded from Java config
Java config name: /etc/krb5.conf
Loaded from Java config
>>> KdcAccessibility: reset
>>> KdcAccessibility: reset
>>>DEBUG <CCacheInputStream>  client principal is [e-mail]
>>>DEBUG <CCacheInputStream> server principal is krbtgt/[e-mail]
>>>DEBUG <CCacheInputStream> key type: 18
>>>DEBUG <CCacheInputStream> auth time: Thu Jun 20 12:46:45 JST 2019
>>>DEBUG <CCacheInputStream> start time: Thu Jun 20 12:46:45 JST 2019
>>>DEBUG <CCacheInputStream> end time: Fri Jun 21 12:46:43 JST 2019
>>>DEBUG <CCacheInputStream> renew_till time: Thu Jun 27 12:46:43 JST 2019
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL; PRE_AUTH;
Found ticket for [e-mail] to go to krbtgt/[e-mail] expiring on Fri Jun 21 12:46:43 JST 2019
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for [e-mail] to go to krbtgt/[e-mail] expiring on Fri Jun 21 12:46:43 JST 2019
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: ........
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbKdcReq send: kdc=kerberos.example.com UDP:88, timeout=30000, number of retries =3, #bytes=661
>>> KDCCommunication: kdc=kerberos.example.com UDP:88, timeout=30000,Attempt =1, #bytes=661
>>> KrbKdcReq send: #bytes read=171
>>> KdcAccessibility: remove kerberos.example.com
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
         cTime is Wed Dec 16 00:15:05 JST 1998 913734905000
         sTime is Thu Jun 20 12:50:30 JST 2019 1561002630000
         suSec is 659395
         error code is 7
         error Message is Server not found in Kerberos database
         cname is [e-mail]
         sname is hive/[e-mail]
         msgType is 30
KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
        at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
        at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
        at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
        at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
        at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)

Issue Links

• is broken by: HIVE-17218 Canonical-ize hostnames for Hive metastore, and HS2 servers.
• is related to: HIVE-22590 Revert HIVE-17218 Canonical-ize hostnames for Hive metastore, and HS2 servers as it causes issues with SSL and LB
• links to: GitHub Pull Request #748
• links to: GitHub Pull Request #3353