[YARN-5302] Yarn Application log Aggregation fails due to NM can not get correct HDFS delegation token II

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: nodemanager, yarn
• Labels: oct16-medium
• Assignee: Xianyin Xin
• Reporter: Xianyin Xin
• Created: 30/Jun/16 12:51
• Updated: 23/Jul/20 11:27

Description

Different with YARN-5098, this happens at NM side. When NM recovers, credentials are read from NMStateStore. When initialize app aggregators, exception happens because of the overdue tokens. The app is a long running service.

LogAggregationService.java

  protected void initAppAggregator(final ApplicationId appId, String user,
      Credentials credentials, ContainerLogsRetentionPolicy logRetentionPolicy,
      Map<ApplicationAccessType, String> appAcls,
      LogAggregationContext logAggregationContext) {

    // Get user's FileSystem credentials
    final UserGroupInformation userUgi =
        UserGroupInformation.createRemoteUser(user);
    if (credentials != null) {
      userUgi.addCredentials(credentials);
    }

   ...

    try {
      // Create the app dir
      createAppDir(user, appId, userUgi);
    } catch (Exception e) {
      appLogAggregator.disableLogAggregation();
      if (!(e instanceof YarnRuntimeException)) {
        appDirException = new YarnRuntimeException(e);
      } else {
        appDirException = (YarnRuntimeException)e;
      }
      appLogAggregators.remove(appId);
      closeFileSystems(userUgi);
      throw appDirException;
    }

Issue Links

• is related to: YARN-5305 Yarn Application Log Aggregation fails due to NM can not get correct HDFS delegation token III
• relates to: YARN-8418 App local logs could leaked if log aggregation fails to initialize for the app
• relates to: YARN-4783 Log aggregation failure for application when Nodemanager is restarted
• relates to: YARN-5098 Yarn Application Log Aggregation fails due to NM can not get correct HDFS delegation token