[YARN-6711] log aggregation can't use other filesystem instead of fs.defaultFS

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.7.1, 3.0.0-alpha3
• Assignee: Unassigned
• Reporter: KWON BYUNGCHANG
• Created: 14/Jun/17 08:33
• Updated: 15/Jun/17 01:12
• Resolved: 15/Jun/17 01:12

Description

hdfs support federated namespace.
fs.defaultFS should choice one of federated namespace.

however if yarn.nodemanager.remote-app-log-dir does not belong to default fs,
nodemanager will stop.

absolutely log-aggregation should can use other namespace instead of default fs.

below is error log

configuration is

fs.defaultFS=hdfs://x3
yarn.nodemanager.remote-app-log-dir=hdfs://x3v/logs

2017-06-14 15:59:47,792 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.IllegalArgumentException: Wrong FS: hdfs://x3v/logs, expected: hdfs://x3
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
    at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:192)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:319)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)
    at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
    at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
    at java.lang.Thread.run(Thread.java:745)
2017-06-14 15:59:47,794 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..

Issue Links

• is duplicated by: YARN-3269 Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path