[YARN-9030] Log aggregation changes to handle filesystems which do not support setting permissions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.2.1
• Component/s: log-aggregation
• Hadoop Flags: Reviewed
• Assignee: Suma Shivaprasad
• Reporter: Suma Shivaprasad
• Created: 16/Nov/18 09:34
• Updated: 04/Sep/19 11:47
• Resolved: 28/Nov/18 23:32

Description

Some cloud storages like ADLS do not support permissions in which case they throw an UnsupportedOperationException. Log aggregation code should log/ignore these exceptions and not set permissions henceforth for log aggregation base dir/sub dirs

2018-11-12 15:37:28,726 WARN  logaggregation.LogAggregationService (LogAggregationService.java:initApp(209)) - Application failed to init aggregation
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to check permissions for dir [abfs://[e-mail]/app-logs]
        at org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController.verifyAndCreateRemoteLogDir(LogAggregationFileController.java:277)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:238)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:204)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:347)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:69)
        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
        at java.lang.Thread.run(Thread.java:748)