[YARN-503] DelegationTokens will be renewed forever if multiple jobs share tokens and the first one sets JOB_CANCEL_DELEGATION_TOKEN to false

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.23.3, 2.0.0-alpha, 3.0.0-alpha1
• Component/s: resourcemanager
• Assignee: Daryn Sharp
• Reporter: Siddharth Seth
• Created: 07/Mar/12 00:30
• Updated: 12/May/16 18:30

Description

The first Job/App to register a token is the one which DelegationTokenRenewer associates with a a specific Token. An attempt to remove/cancel these shared tokens by subsequent jobs doesn't work - since the JobId will not match.
As a result, Even if subsequent jobs have MRJobConfig.JOB_CANCEL_DELEGATION_TOKEN set to true - tokens will not be cancelled when those jobs complete.
Tokens will eventually be removed from the RM / JT when the service that issued them considers them to have expired or via an explicit cancelDelegationTokens call (not implemented yet in 23).
A side affect of this is that the same delegation token will end up being renewed multiple times (a separate TimerTask for each job which uses the token).

DelegationTokenRenewer could maintain a reference count/list of jobIds for shared tokens.

Issue Links

• is part of: MAPREDUCE-3101 [Umbrella] Security issues in YARN
• is required by: MAPREDUCE-5096 Deprecate ability to not cancel tokens when job completes