[FLINK-9683] HistoryServer should honor fs.default-scheme

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.2
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available starter
• Assignee: Jiang Xin
• Reporter: Ethan Li
• Created: 27/Jun/18 19:02
• Updated: 03/Feb/21 01:22
• Resolved: 02/Feb/21 14:11

Description

I am using release-1.4.2,

 

With fs.default-scheme and  fs.hdfs.hadoopconf set correctly, 

when setting 

historyserver.archive.fs.dir: /tmp/flink/cluster-name/jmarchive

I am seeing

2018-06-27 18:51:12,692 WARN org.apache.flink.runtime.webmonitor.history.HistoryServer - Failed to create Path or FileSystem for directory '/tmp/flink/cluster-name/jmarchive'. Directory will not be monitored.
java.lang.IllegalArgumentException: The scheme (hdfs://, file://, etc) is null. Please specify the file system scheme explicitly in the URI.
at org.apache.flink.runtime.webmonitor.WebMonitorUtils.validateAndNormalizeUri(WebMonitorUtils.java:300)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:168)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:113)
at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.main(HistoryServer.java:110)

 

And then if I set 

historyserver.archive.fs.dir: hdfs:///tmp/flink/cluster-name/jmarchive

I am seeing:

 

java.io.IOException: The given file system URI (hdfs:///tmp/flink/cluster-name/jmarchive) did not describe the authority (like for example HDFS NameNode address/port or S3 host). The attempt to use a configured default authority failed: Hadoop configuration for default file system ('fs.default.name' or 'fs.defaultFS') contains no valid authority component (like hdfs namenode, S3 host, etc)
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:169)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)

The only way it works is to provide the full path of hdfs like:

#historyserver.archive.fs.dir: hdfs://<hdfs-cluster>/tmp/flink/cluster-name/jmarchive

 

Above situations are because there are two parts of code treating "scheme" differently. 

https://github.com/apache/flink/blob/release-1.4.2/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorUtils.java#L299-L302

https://github.com/apache/flink/blob/release-1.4.2/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L335-L338

I believe the first case should be supported if users have set fs.default-scheme 

Issue Links

• links to: GitHub Pull Request #14771