Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9519

TFile log aggregation file format is not working for yarn.log-aggregation.TFile.remote-app-log-dir config

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.3.0, 3.2.1, 3.1.3
    • Component/s: log-aggregation
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The TFile log aggregation file format is not sensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config.

      In LogAggregationTFileController$initInternal:

          this.remoteRootLogDir = new Path(
              conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
                  YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
      

      So the remoteRootLogDir is only aware of the yarn.nodemanager.remote-app-log-dir config, while other file format, like IFile defaults to the file format config, so its priority is higher.

      From LogAggregationIndexedFileController$initInternal:

          String remoteDirStr = String.format(
              YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT,
              this.fileControllerName);
          String remoteDir = conf.get(remoteDirStr);
          if (remoteDir == null || remoteDir.isEmpty()) {
            remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
                YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR);
          }
      

      (Where these configs are: )

      public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT
            = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir";
      public static final String NM_REMOTE_APP_LOG_DIR = 
          NM_PREFIX + "remote-app-log-dir";
      

      I suggest TFile should try to obtain the remote dir config from yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not specified falls back to the yarn.nodemanager.remote-app-log-dir config.

        Attachments

        1. YARN-9519.005.patch
          21 kB
          Adam Antal
        2. YARN-9519.004.patch
          21 kB
          Adam Antal
        3. YARN-9519.003.patch
          21 kB
          Adam Antal
        4. YARN-9519.002.patch
          19 kB
          Adam Antal
        5. YARN-9519.001.patch
          5 kB
          Adam Antal

          Issue Links

            Activity

              People

              • Assignee:
                adam.antal Adam Antal
                Reporter:
                adam.antal Adam Antal
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: