[YARN-9542] Fix LogsCLI guessAppOwner ignores custom file format suffix

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.1.2
• Fix Version/s: 3.2.1, 3.1.3
• Component/s: log-aggregation
• Assignee: Prabhu Joseph
• Reporter: Prabhu Joseph
• Created: 10/May/19 10:32
• Updated: 31/May/21 09:49
• Resolved: 29/May/19 22:07

Description

LogsCLI guessAppOwner ignores custom file format suffix yarn.log-aggregation.%s.remote-app-log-dir-suffix / Default IndexedFileController Suffix (

{yarn.nodemanager.remote-app-log-dir-suffix}

-ifile or logs-ifile). It considers only yarn.nodemanager.remote-app-log-dir-suffix or default logs.

Repro:

yarn-site.xml

yarn.log-aggregation.file-formats ifile
yarn.log-aggregation.file-controller.ifile.class org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
yarn.log-aggregation.ifile.remote-app-log-dir app-logs
yarn.resourcemanager.connect.max-wait.ms 1000

core-site.xml:

ipc.client.connect.max.retries 3
ipc.client.connect.retry.interval 10

Run a Job with above configs and Stop the RM.

[ambari-qa@yarn-ats-1 ~]$ yarn logs -applicationId application_1557482389195_0001
2019-05-10 10:03:58,215 INFO client.RMProxy: Connecting to ResourceManager at yarn-ats-1/172.26.81.91:8050
Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
Can not find the appOwner. Please specify the correct appOwner
Could not locate application logs for application_1557482389195_0001


[ambari-qa@yarn-ats-1 ~]$ hadoop fs -ls /app-logs/ambari-qa/logs-ifile/application_1557482389195_0001
Found 1 items
-rw-r-----   3 ambari-qa supergroup      18058 2019-05-10 10:01 /app-logs/ambari-qa/logs-ifile/application_1557482389195_0001/yarn-ats-1_45454

Issue Links

• Is contained by: YARN-9558 Log Aggregation testcases failing
• is required by: YARN-6929 yarn.nodemanager.remote-app-log-dir structure is not scalable