[YARN-9519] TFile log aggregation file format is not working for yarn.log-aggregation.TFile.remote-app-log-dir config

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0, 3.2.1, 3.1.3
• Component/s: log-aggregation
• Target Version/s: 3.3.0
• Hadoop Flags: Reviewed
• Assignee: Adam Antal
• Reporter: Adam Antal
• Created: 29/Apr/19 12:35
• Updated: 17/May/19 08:34
• Resolved: 14/May/19 17:51

Description

The TFile log aggregation file format is not sensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config.

In LogAggregationTFileController$initInternal:

    this.remoteRootLogDir = new Path(
        conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
            YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));

So the remoteRootLogDir is only aware of the yarn.nodemanager.remote-app-log-dir config, while other file format, like IFile defaults to the file format config, so its priority is higher.

From LogAggregationIndexedFileController$initInternal:

    String remoteDirStr = String.format(
        YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT,
        this.fileControllerName);
    String remoteDir = conf.get(remoteDirStr);
    if (remoteDir == null || remoteDir.isEmpty()) {
      remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
          YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR);
    }

(Where these configs are: )

public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT
      = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir";
public static final String NM_REMOTE_APP_LOG_DIR = 
    NM_PREFIX + "remote-app-log-dir";

I suggest TFile should try to obtain the remote dir config from yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not specified falls back to the yarn.nodemanager.remote-app-log-dir config.

Issue Links

• is related to: YARN-9558 Log Aggregation testcases failing
• is required by: YARN-6929 yarn.nodemanager.remote-app-log-dir structure is not scalable