[HADOOP-7967] Need generalized multi-token filesystem support

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.23.1, 0.24.0
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: fs, security
• Target Version/s: 0.23.3, 2.0.0-alpha
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 10/Jan/12 17:06
• Updated: 11/Oct/12 17:45
• Resolved: 20/Aug/12 15:36

Description

Multi-token filesystem support and its interactions with the MR TokenCache is problematic. The TokenCache tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently FileSystem will leak tokens even when some tokens are already present.

The decision for token acquisition, and which tokens, should be pushed all the way down into the FileSystem level. The TokenCache should be ignorant and simply request tokens from each FileSystem.

Issue Links

• blocks: MAPREDUCE-3825 MR should not be getting duplicate tokens for a MR Job.
• is duplicated by: HDFS-2854 SecurityUtil.buildTokenService returns java.net.UnknownHostException when using paths like viewfs://default/some/path
• is related to: MAPREDUCE-3639 TokenCache likely broken for FileSystems which don't issue delegation tokens
• is related to: HADOOP-7933 Viewfs changes for MAPREDUCE-3529
• is related to: HDFS-2710 HDFS part of MAPREDUCE-3529, HADOOP-7933
• relates to: HDFS-2854 SecurityUtil.buildTokenService returns java.net.UnknownHostException when using paths like viewfs://default/some/path
• relates to: HADOOP-8701 Reduce visibility of getDelegationToken
• relates to: HADOOP-8048 Allow merging of Credentials