[HADOOP-10453] Do not use AuthenticatedURL in hadoop core

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Haohui Mai
• Created: 31/Mar/14 18:36
• Updated: 07/Jan/17 01:56

Description

As daryn has suggested in HDFS-4564:

AuthenticatedURL is not used because it is buggy in part to causing replay attacks, double attempts to kerberos authenticate with the fallback authenticator if the TGT is expired, incorrectly uses the fallback authenticator (required by oozie servers) to add the username parameter which webhdfs has already included in the uri.

AuthenticatedURL's attempt to do SPNEGO auth is a no-op because the JDK transparently does SPNEGO when the user's Subject (UGI) contains kerberos principals. Since AuthenticatedURL is now not used, webhdfs has to check the TGT itself for token operations.
Bottom line is AuthenticatedURL is unnecessary and introduces nothing but problems for webhdfs. It's only useful for oozie's anon/non-anon support.

However, several functionalities that relies on SPNEGO in secure mode suffer from the same problem. For example, NNs / JNs create HTTP connections to exchange fsimage and edit logs. Currently all of them are through AuthenticatedURL. This needs to be fixed to avoid security vulnerabilities.

This jira purposes to remove AuthenticatedURL from hadoop core and to move it to oozie.

Issue Links

• is related to: HADOOP-10799 Refactor HTTP delegation token logic from httpfs into reusable code in hadoop-common.
• relates to: HDFS-4564 Webhdfs returns incorrect http response codes for denied operations
• relates to: HADOOP-12787 KMS SPNEGO sequence does not work with WEBHDFS