Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5302

Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: nodemanager, yarn
    • Labels:

      Description

      Different with YARN-5098, this happens at NM side. When NM recovers, credentials are read from NMStateStore. When initialize app aggregators, exception happens because of the overdue tokens. The app is a long running service.

      LogAggregationService.java
        protected void initAppAggregator(final ApplicationId appId, String user,
            Credentials credentials, ContainerLogsRetentionPolicy logRetentionPolicy,
            Map<ApplicationAccessType, String> appAcls,
            LogAggregationContext logAggregationContext) {
      
          // Get user's FileSystem credentials
          final UserGroupInformation userUgi =
              UserGroupInformation.createRemoteUser(user);
          if (credentials != null) {
            userUgi.addCredentials(credentials);
          }
      
         ...
      
          try {
            // Create the app dir
            createAppDir(user, appId, userUgi);
          } catch (Exception e) {
            appLogAggregator.disableLogAggregation();
            if (!(e instanceof YarnRuntimeException)) {
              appDirException = new YarnRuntimeException(e);
            } else {
              appDirException = (YarnRuntimeException)e;
            }
            appLogAggregators.remove(appId);
            closeFileSystems(userUgi);
            throw appDirException;
          }
      

        Attachments

        1. YARN-5032.001.patch
          16 kB
          Xianyin Xin
        2. YARN-5032.002.patch
          15 kB
          Xianyin Xin
        3. YARN-5302.003.patch
          16 kB
          Xianyin Xin
        4. YARN-5302.004.patch
          15 kB
          Xianyin Xin

          Issue Links

            Activity

              People

              • Assignee:
                xinxianyin Xianyin Xin
                Reporter:
                xinxianyin Xianyin Xin
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: