Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5302

Yarn Application log Aggregation fails due to NM can not get correct HDFS delegation token II

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • nodemanager, yarn

    Description

      Different with YARN-5098, this happens at NM side. When NM recovers, credentials are read from NMStateStore. When initialize app aggregators, exception happens because of the overdue tokens. The app is a long running service.

      LogAggregationService.java
        protected void initAppAggregator(final ApplicationId appId, String user,
            Credentials credentials, ContainerLogsRetentionPolicy logRetentionPolicy,
            Map<ApplicationAccessType, String> appAcls,
            LogAggregationContext logAggregationContext) {
      
          // Get user's FileSystem credentials
          final UserGroupInformation userUgi =
              UserGroupInformation.createRemoteUser(user);
          if (credentials != null) {
            userUgi.addCredentials(credentials);
          }
      
         ...
      
          try {
            // Create the app dir
            createAppDir(user, appId, userUgi);
          } catch (Exception e) {
            appLogAggregator.disableLogAggregation();
            if (!(e instanceof YarnRuntimeException)) {
              appDirException = new YarnRuntimeException(e);
            } else {
              appDirException = (YarnRuntimeException)e;
            }
            appLogAggregators.remove(appId);
            closeFileSystems(userUgi);
            throw appDirException;
          }
      

      Attachments

        1. YARN-5302.004.patch
          15 kB
          Xianyin Xin
        2. YARN-5302.003.patch
          16 kB
          Xianyin Xin
        3. YARN-5032.002.patch
          15 kB
          Xianyin Xin
        4. YARN-5032.001.patch
          16 kB
          Xianyin Xin

        Issue Links

          Activity

            People

              xinxianyin Xianyin Xin
              xinxianyin Xianyin Xin
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: