[YARN-8279] AggregationLogDeletionService does not honor yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Workaround
Affects Version/s: 2.9.1
Fix Version/s: None
Component/s: log-aggregation
Labels:
None

Release Note:

Hide
This issue affects only when IndexedFormat is used for log-aggregation by setting below properties:
yarn.log-aggregation.file-formats=IndexedFormat
yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController

The fix is to set the below properties in yarn-site.xml and then restart Mapreduce HistoryServer and YARN services.
yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix and yarn.nodemanager.remote-app-log-dir-suffix to same value "logs-ifile" which serves AggregationLogDeletionService

Show
This issue affects only when IndexedFormat is used for log-aggregation by setting below properties: yarn.log-aggregation.file-formats=IndexedFormat yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController The fix is to set the below properties in yarn-site.xml and then restart Mapreduce HistoryServer and YARN services. yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix and yarn.nodemanager.remote-app-log-dir-suffix to same value "logs-ifile" which serves AggregationLogDeletionService

Description

AggregationLogDeletionService does not honor yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix. AggregationLogService writes the logs into /app-logs/<user-dir>/logs-ifile where as AggregationLogDeletion tries to delete from /app-logs/<user-dir>/logs.

Workaround is to set yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix and yarn.nodemanager.remote-app-log-dir-suffix to same value "logs-ifile" and Restart HistoryServer which serves AggregationLogDeletionService

AggregationLogDeletionService has to check the format and based upon that choose the suffix. Currently it only checks the older suffix yarn.nodemanager.remote-app-log-dir-suffix.

AggregatedLogDeletionService tries to delete older suffix directory.

2018-05-11 08:48:19,989 ERROR logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:logIOException(182)) - Could not read the contents of hdfs://prabhucluster:8020/app-logs/hive/logs
java.io.FileNotFoundException: File hdfs://prabhucluster:8020/app-logs/hive/logs does not exist.
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:923)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:985)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:981)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:992)
        at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.deleteOldLogDirsFrom(AggregatedLogDeletionService.java:98)
        at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:85)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

Attachments

Activity

Comment

This comment will be Viewable by All Users Viewable by All Users

Cancel

People

Assignee:: Tarun Parimi

Reporter:: Prabhu Joseph

Votes:: 1 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 11/May/18 11:50

Updated:: 20/Nov/18 20:09

Resolved:: 11/Oct/18 09:31

Agile

View on Board

AggregationLogDeletionService does not honor yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix