[IMPALA-6726] Catalog server's kerberos ticket gets deleted after 'ticket_lifetime' on SLES11

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0, Impala 2.12.0
• Fix Version/s: Impala 2.12.0
• Component/s: Security
• Labels: kerberos security
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-9
• Assignee: Michael Ho
• Reporter: Sailesh Mukil
• Created: 23/Mar/18 00:34
• Updated: 30/Aug/18 18:28
• Resolved: 29/Mar/18 15:33

Description

On SLES11, it was noticed that after 'ticket_lifetime', the kerberos ticket gets deleted by the Java krb5 library. mikesbrown noticed this from 2.11, and we confirmed that it shows up in 2.12 as well.

I turned on the Java kerberos debug logging and found this in the log messages:

W0322 07:51:43.617998 12118 UserGroupInformation.java:1403] Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1521730246019
>>>DEBUG <CCacheInputStream>  client principal is impala/[e-mail]
>>>DEBUG <CCacheInputStream> server principal is krbtgt/[e-mail]
>>>DEBUG <CCacheInputStream> key type: 16
>>>DEBUG <CCacheInputStream> auth time: Thu Mar 22 07:21:58 PDT 2018
>>>DEBUG <CCacheInputStream> start time: Thu Mar 22 07:51:46 PDT 2018
>>>DEBUG <CCacheInputStream> end time: Thu Mar 22 07:51:58 PDT 2018
>>>DEBUG <CCacheInputStream> renew_till time: Thu Mar 22 07:51:58 PDT 2018
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL;
Found ticket for impala/[e-mail] to go to krbtgt/[e-mail] expiring on Thu Mar 22 07:51:58 PDT 2018
Removed and destroyed the expired Ticket
Destroyed KerberosTicket
W0322 07:52:04.195199 12201 UserGroupInformation.java:1920] PriviledgedActionException as:impala/[e-mail] (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0322 07:52:04.200016 12201 UserGroupInformation.java:1403] Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1521730306038

The backend ticket acquisition thread however keeps running and claiming to have re-acquired a ticket every 'ticket_lifetime' period.

I tried turning off the 'use_kudu_kinit' flag and this bug didn't show up in that mode.

Still investigating the bug.

Issue Links

• is blocked by: KUDU-2385 Kerberos ticket reacquisition with Heimdal krb5 likely broken