-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.2.0
-
Component/s: log-aggregation
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
The TFile log aggregation file format is not sensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config.
In LogAggregationTFileController$initInternal:
this.remoteRootLogDir = new Path( conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
So the remoteRootLogDir is only aware of the yarn.nodemanager.remote-app-log-dir config, while other file format, like IFile defaults to the file format config, so its priority is higher.
From LogAggregationIndexedFileController$initInternal:
String remoteDirStr = String.format( YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT, this.fileControllerName); String remoteDir = conf.get(remoteDirStr); if (remoteDir == null || remoteDir.isEmpty()) { remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR); }
(Where these configs are: )
public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir"; public static final String NM_REMOTE_APP_LOG_DIR = NM_PREFIX + "remote-app-log-dir";
I suggest TFile should try to obtain the remote dir config from yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not specified falls back to the yarn.nodemanager.remote-app-log-dir config.