[YARN-2790] NM can't aggregate logs past HDFS delegation token expiry.

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.6.0
• Component/s: nodemanager
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Jian He
• Reporter: Tassapol Athiapinya
• Created: 31/Oct/14 19:12
• Updated: 01/Dec/14 03:08
• Resolved: 01/Nov/14 23:36

Description

We shorten hdfs delegation token lifetime, set RM to act as as a proxy user in order to be able to renew the token on behalf of a user submitting the application. Still we see NM log aggregation fail due to token expiry error.

2014-10-31 00:11:56,579 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(481)) - Application just finished : application_1414705229376_0004
2014-10-31 00:11:56,588 WARN  ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 75 for user1) is expired
2014-10-31 00:11:56,589 ERROR logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainers(233)) - Cannot create writer for app application_1414705229376_0004. Skip log upload this time.

Issue Links

• is broken by: YARN-2704 Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time