Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6711

log aggregation can't use other filesystem instead of fs.defaultFS

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.7.1, 3.0.0-alpha3
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      hdfs support federated namespace.
      fs.defaultFS should choice one of federated namespace.

      however if yarn.nodemanager.remote-app-log-dir does not belong to default fs,
      nodemanager will stop.

      absolutely log-aggregation should can use other namespace instead of default fs.

      below is error log

      configuration is

      fs.defaultFS=hdfs://x3
      yarn.nodemanager.remote-app-log-dir=hdfs://x3v/logs

      2017-06-14 15:59:47,792 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
      java.lang.IllegalArgumentException: Wrong FS: hdfs://x3v/logs, expected: hdfs://x3
          at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)
          at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
          at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
          at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
          at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
          at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
          at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:192)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:319)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)
          at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
          at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
          at java.lang.Thread.run(Thread.java:745)
      2017-06-14 15:59:47,794 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                magnum KWON BYUNGCHANG
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: