[MAPREDUCE-3825] MR should not be getting duplicate tokens for a MR Job.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.23.1, 2.0.0-alpha
• Component/s: security
• Target Version/s: 0.23.3, 2.0.0-alpha
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 06/Feb/12 23:23
• Updated: 10/Mar/15 04:31
• Resolved: 16/Apr/14 22:04

Description

This is the counterpart to HADOOP-7967.
MR gets tokens for all input, output and the default filesystem when a MR job is submitted.

The APIs in FileSystem make it challenging to avoid duplicate tokens when there are file systems that have embedded
filesystems.

Here is the original description that Daryn wrote:
The token cache currently tries to assume a filesystem's token service key. The assumption generally worked while there was a one to one mapping of filesystem to token. With the advent of multi-token filesystems like viewfs, the token cache will try to use a service key (ie. for viewfs) that will never exist (because it really gets the mounted fs tokens).

The descriop

Issue Links

• is blocked by: HADOOP-7967 Need generalized multi-token filesystem support
• relates to: HDFS-2854 SecurityUtil.buildTokenService returns java.net.UnknownHostException when using paths like viewfs://default/some/path
• relates to: HADOOP-8048 Allow merging of Credentials
• relates to: MAPREDUCE-3849 Change TokenCache's reading of the binary token file
• relates to: MAPREDUCE-3850 Avoid redundant calls for tokens in TokenCache