[YARN-2704] Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Jian He
• Reporter: Jian He
• Created: 17/Oct/14 18:23
• Updated: 09/Apr/15 21:50
• Resolved: 27/Oct/14 22:52

Description

In secure mode, YARN requires the hdfs-delegation token to do localization and log aggregation on behalf of the user. But the hdfs delegation token will eventually expire after max-token-life-time. So, localization and log aggregation will fail after the token expires.

Issue Links

• breaks: YARN-2964 RM prematurely cancels tokens for jobs that submit jobs (oozie)
• breaks: YARN-3439 RM fails to renew token when Oozie launcher leaves before sub-job finishes
• breaks: YARN-2790 NM can't aggregate logs past HDFS delegation token expiry.
• duplicates: YARN-2425 When Application submitted by via Yarn RM WS, log aggregation does not happens
• is related to: YARN-2794 Fix log msgs about distributing system-credentials
• is related to: YARN-3472 Possible leak in DelegationTokenRenewer#allTokens
• is related to: HDFS-7295 Support arbitrary max expiration times for delegation token