Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9030

Log aggregation changes to handle filesystems which do not support setting permissions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.2.1
    • Component/s: log-aggregation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Some cloud storages like ADLS do not support permissions in which case they throw an UnsupportedOperationException. Log aggregation code should log/ignore these exceptions and not set permissions henceforth for log aggregation base dir/sub dirs

      2018-11-12 15:37:28,726 WARN  logaggregation.LogAggregationService (LogAggregationService.java:initApp(209)) - Application failed to init aggregation
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to check permissions for dir [abfs://testctr1@test.blob.core.windows.net/app-logs]
              at org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController.verifyAndCreateRemoteLogDir(LogAggregationFileController.java:277)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:238)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:204)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:347)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:69)
              at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
              at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
              at java.lang.Thread.run(Thread.java:748)
      
      

        Attachments

        1. YARN-9030.2.patch
          6 kB
          Suma Shivaprasad
        2. YARN-9030.1.patch
          5 kB
          Suma Shivaprasad

          Activity

            People

            • Assignee:
              suma.shivaprasad Suma Shivaprasad
              Reporter:
              suma.shivaprasad Suma Shivaprasad
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: