[IMPALA-9359] Recover gracefully from corrupt kerberos credential cache

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 3.4.0
• Component/s: Security
• Labels: kerberos
• Epic Color: ghx-label-8
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 05/Feb/20 01:48
• Updated: 23/Dec/20 21:43
• Resolved: 12/Mar/20 04:48

Description

1. Start up a kerberized Impala cluster
2. Corrupt the kerberos ticket cache used by impala /tmp/krb5cc_impala_internal
3. Observe queries fail. The details depend a lot on timing, etc. I have seen communication failures between impalads and with other systems, e.g. HDFS.
4. The system will stay wedge in this state indefinitely

We have seen this happen once in production from /tmp filling up.

I prototyped a fix that amounts to re-running Kinit() to blow away the broken credential cache. It needs more work to be production-ready

Issue Links

• depends upon: KUDU-3050 Recover gracefully from corrupt kerberos credential cache
• is duplicated by: IMPALA-5385 Print appropriate error for failed TGT creation due to /tmp being full
• is related to: IMPALA-9367 Allow specifying MEMORY kerberos credential cache