[HADOOP-12649] Improve Kerberos diagnostics and failure handling

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.1
• Component/s: security

Environment

Kerberos

• Assignee: Unassigned Assign to me
• Reporter: Steve Loughran
• Created: 16/Dec/15 13:11
• Updated: 29/Nov/18 14:39

Description

Sometimes —apparently— some people cannot get kerberos to work.

The ability to diagnose problems here is hampered by some aspects of UGI

1. the only way to turn on JAAS debug information is through an env var, not within the JVM
2. failures are potentially underlogged
3. exceptions raised are generic IOEs, so can't be trapped and filtered
4. failure handling on the TGT renewer thread is nonexistent
5. the code is barely-readable, underdocumented mess.

Issue Links

• depends upon: HADOOP-12751 While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
• depends upon: HADOOP-12906 AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.
• depends upon: HADOOP-13056 Print expected values when rejecting a server's determined principal
• depends upon: HADOOP-10315 Log the original exception when getGroups() fail in UGI.
• depends upon: HADOOP-10523 Hadoop services (such as RM, NN and JHS) throw confusing exception during token auto-cancelation
• depends upon: HADOOP-14327 KerberosAuthenticationHandler#authenticate throws meaningless exception when server principals set is empty
• depends upon: HADOOP-12741 UserGroupInformation.loginUserFromKeytab() creates background thread which is not getting killed even after application exited
• depends upon: HADOOP-12563 Updated utility to create/modify token files
• depends upon: HADOOP-12510 Need improved WARN or ERROR when token based auth fails for kmsclient request
• depends upon: HADOOP-13604 Abort retry loop when RPC has an unrecoverable Auth error
• depends upon: HADOOP-12664 UGI auto-renewer does not verify kinit availability during initialization
• depends upon: HADOOP-10776 Open up already widely-used APIs for delegation-token fetching & renewal to ecosystem projects
• depends upon: HDFS-9732 Improve DelegationTokenIdentifier.toString() for better logging
• depends upon: YARN-4682 AMRM client to log when AMRM token updated
• depends upon: YARN-4721 RM to try to auth with HDFS on startup, retry with max diagnostics on failure
• depends upon: HADOOP-11404 Clarify the "expected client Kerberos principal is null" authorization message
• depends upon: HADOOP-15959 revert HADOOP-12751
• depends upon: HADOOP-8787 KerberosAuthenticationHandler should include missing property names in configuration
• is depended upon by: HADOOP-12426 Add Entry point for Kerberos health check
• is related to: HADOOP-12770 KMSClientProvider addDelegationTokens won't add if the credentials contain an expired one
• is related to: HADOOP-12650 Document all of the secret env vars
• is related to: YARN-4629 Distributed shell breaks under strong security
• relates to: SLIDER-993 Client fails to start ZK Registry client: Entry "Client" not found; JAAS config =
• relates to: SLIDER-1027 add a kdiag command for kerberos diagnostics
• relates to: ZOOKEEPER-2344 Provide more diagnostics/stack traces on SASL Auth failure
• relates to: HADOOP-13590 Retry until TGT expires even if the UGI renewal thread encountered exception
• relates to: SLIDER-1035 Kdiag enhancements
• relates to: YARN-4653 Document YARN security model from the perspective of Application Developers

Sub-Tasks

• pull out logic to determine principal, allow kdiag to get at it directly: Open
• Improve diagnostics/use of envvar/sysprop credential propagation: Resolved
• Client.handleSaslConnectionFailure() uses wrong user in exception text: Resolved
• KDiag to look at HADOOP_TOKEN_FILE_LOCATION more: Open
• Credentials to include text of inner IOE when rethrowing wrapped: Open
• UGI to log@ debug stack traces when failing to find groups for a user: Resolved
• Make kdiag something services can use directly on startup: Patch Available
• kdiag to add a --DEFAULTREALM option: Open
• Drop the @LimitedPrivate maker off UGI, as its clearly untrue: Resolved
• kdiag to add a --url option to attempt a SPNEGO-authed GET: Open
• Have an explicit KerberosAuthException for UGI to throw, text from public constants: Resolved
• replace `Can't get Master Kerberos principal` message with useful information about principal and configuration item: Open
• client.handleSaslConnectionFailure needlessly wraps IOEs: Patch Available
• Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds: Resolved
• KDiag to add cli options for proxies and taking to HDFS for listing & DT retrieval: Open
• UGI.createLoginUser to log token filename & token identifiers on load: Open